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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  has  been  actively  involved  in  the 
development  of  After  Action  Review  (AAR)  methods  and  tools  for 
the  live,  virtual,  and  constructive  training  environments 
beginning  with  the  development  of  tactical  engagement  simulation 
in  the  mid-seventies.  As  part  of  the  Army's  Warfighter  XXI 
program,  the  U.S.  Army  Training  and  Doctrine  Command  is 
attempting  to  develop  a  Standard  Army  After  Action  Review  System 
(STAARS)  for  use  across  training  environments  that  will  help 
guarantee  a  certain  level  of  AAR  quality,  reduce  costly 
duplication  in  AAR  system  development,  and  provide  exercise  data 
to  support  advanced  concepts  requirements  and  research, 
development,  and  acquisition  efforts.  At  the  same  time,  the 
Simulation,  Training,  and  Instrumentation  Command  has  formed  an 
AAR  Integrated  Product  Team  to  leverage  AAR  system  development 
efforts . 

This  report  defines  major  variables  to  be  considered  in 
developing  a  STAARS,  describes  the  state  of  the  art  and  lessons 
learned  in  terms  of  AAR  system  capabilities,  recommends 
specifications  for  a  STAARS,  and  identifies  behavioral  and 
technological  research  and  development  efforts  needed  to  support 
implementation  of  the  STAARS  concept.  Most  of  the  recommended 
specifications  have  been  included  in  the  STAARS  Operational 
Requirements  Document. 

The  work  described  in  this  report  is  a  portion  of  research 
task  2114,  SYNTRAIN:  Distributed  Interactive  Simulation  Systems. 
This  task  supports  a  Memorandum  of  Agreement  entitled  "Training 
Research  Support  of  Combined  Arms  Tactical  Trainer  Development 
Efforts,"  signed  24  February  1993.  Parties  to  this  agreement  are 
the  U.S.  Army  Project  Manager  for  Combined  Arms  Tactical  Trainer 
and  ARI . 


A  M.  SIMUTIS 


Deputy  Director 
(Science  and  Technology) 


EDGAR  M."  JOHNSON 
Director 
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ST7\NDARDIZING  ARMY  AFTER  ACTION  REVIEW  SYSTEMS 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U,S.  Army  is  preparing  to  develop  a  Standardized  Army 
After  Action  Review  System  (STAARS)  that  can  be  used  within  and 
across  live,  virtual,  and  constructive  training  environments. 
There  is  a  need  to  define  STAARS  capabilities  for  a  STAARS 
operational  requirements  document,  and  there  is  a  need  to  define 
technical  and  behavioral  research  and  development  efforts 
necessary  to  make  sure  the  technology  base  can  support  the  STAARS 
concept . 

ARI  has  had  a  long  involvement  in  the  development  of  AAR 
procedures  and  systems  that  includes  two  ongoing  efforts  to 
develop  After  Action  Review  (AAR)  systems  for  the  virtual 
environment  and  the  organization  of  Army-wide  AAR  conferences.  A 
review  of  published  and  unpublished  findings  of  these  efforts, 
combined  with  a  review  of  literature  relevant  to  other  AAR 
systems,  were  examined  to  identify  capabilities  critical  to  a 
STAARS  and  to  identify  research  and  development  issues  that  need 
to  be  addressed  to  support  the  STAARS  concept.  Topics  addressed 
included  adequacy  of  the  variety  of  data  displays  available  to 
support  AARs,  tools  to  help  trainers  conduct  an  AAR,  tools  to 
help  trainers  prepare  Take  Home  Packages  to  supplement  and 
reinforce  teaching  points  from  AARs,  archiving  of  exercise  data 
to  support  research  applications,  and  human- computer  interface 
issues . 

Findings : 

Substantial  progress  has  been  made  designing  AAR  aids  that 
can  help  trainers  and  analysts  examine  the  performance  of  units 
more  objectively,  but  there  are  still  complex  unit  behaviors 
calling  for  innovative  AAR  aids.  It  is  imperative  that  a  STAARS 
provide  users  with  the  capability  to  implement  new  types  of  AAR 
aids  integrating  planning  data,  terrain  data,  communications 
data,  behavioral  observation  data,  and  electronic  data  streams. 

Standardizing  AAR  aids  within  echelons  across  training 
environments  helps  to  ensure  that  the  information  contained  in 
the  aids  will  be  readily  interpretable  to  users,  but  there  are 
also  drawbacks  to  this  approach.  By  limiting  the  exact  design 
features  of  data  displays  for  each  echelon  to  those  supported  by 
data  elements  common  to  all  three  environments,  the  U.S.  Army 
will  throw  out  many  of  the  benefits  gained  by  moving  toward 
electronic  battlefields.  STAARS  should  allow  users  to  store  and 
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analyze  all  electronic  data  produced  within  any  of  the  three 
training  environments. 

Providing  AARs  at  lower  echelons  shortly  after  the  end  of  an 
exercise  is  a  major  challenge  for  an  AAR  system,  and  meeting  this 
challenge  requires  automation  of  the  AAR  aids  preparation 
process.  Multiple  AAR  systems  have  been  developed  that  analyze 
the  network  data  stream  to  support  the  automatic  generation  of 
AAR  aids,  but  there  are  many  tactical  events  critical  to  the 
timing  of  AAR  aid  production  that  can  only  be  recognized  by  a 
human  being.  At  least  one  AAR  system  has  demonstrated  that 
automated  AAR  aid  production  can  be  accomplished  using  a  mix  of 
electronic  data  stream  analysis  and  trainer  responses  to  on¬ 
screen  prompts.  This  latter  system  has  also  demonstrated  the 
capability  to  move  back  in  history  and  create  AAR  aids  while  it 
continues  to  collect  exercise  data.  Automation  must  be  applied 
carefully  to  the  AAR  aid  preparation  process  because  it  can  have 
the  effect  of  distracting  trainers  from  their  exercise  control 
functions . 

Utilization  of  Findings: 

Many  of  the  findings  from  this  research  were  used  at  the 
National  Simulation  Center  as  capability  specifications  and 
rationale  for  the  STAARS  Operational  Requirements  Document. 

These  findings  are  also  being  used  by  the  U.S.  Army  Simulation, 
Training,  and  Instrumentation  Command's  AAR  Integrated  Product 
Team  to  help  integrate  AAR  system  development  efforts. 
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Introduction 


Thfi  After  Action  Review  (AAR)  Process. 

The  AAR  is  the  Array's  approach  for  providing  feedback  to 
units  after  collective  training  exercises.  It  is  an  interactive 
process  in  which  exercise  participants  discuss  mission  planning 
and  execution  under  the  guidance  of  a  trainer  (Scott,  1983) .  The 
starting  point  for  the  AAR  is  normally  a  description  of  the 
unit's  plans  for  the  mission  followed  by  a  discussion  of  what 
happened  during  the'  mission  (Department  of  the  Army,  1993)  .  The 
discussions  might  be  guided  in  part,  through  the  use  of  data 
displays  illustrating  what  happened  during  an  exercise.  The  goal 
of  the  AAR  is  to  identify  unit  strengths  and  weaknesses  in  enough 
detail  to  point  the  way  towards  possible  corrective  actions 
(Downs,  Johnson, &  Fallesen,  1987). 

An  example  of  how  interactive  discussions  might  result  in  the 
identification  of  concrete  corrective  actions  is  as  follows.  A 
battalion  task  force  sustained  heavy  losses  and  failed  to  hold 
its  position  in  performing  a  defensive  mission.  The  use  of 
minefields  and  terrain  features  to  force  the  enemy  to  move  in  an 
area  covered  by  direct  and  supporting  fires  was  a  key  part  of  the 
unit's  plan  for  the  mission,  but  no  fires  were  used  during  a 
thirty  minute  period  when  the  enemy' s  main  body  halted  to  breach 
a  minefield.  Discussions  of  what  participants  remembered  about ^ 
the  exercise  might  reveal  that  the  company  team  responsible  for 
covering  the  minefield  with  direct  fire  was  not  in  a  position  to 
observe  it.  Further  discussions  might  reveal  inadequate  early 
coordination  between  the  company  teams  and  the  engineers 
responsible  for  emplacing  the  minefield.  The  "corrective  action 
might  involve  the  company  team  commander  and  the  engineer  platoon 
leader  developing  and  applying  a  checklist  of  all  the  points  on 
which  they  should  coordinate . 

I 

The  AAR  process  is  intended  to  apply  to  live,  virtual, 
constructive,  and  mixed  environment  exercises.  A  live  exercise 
is  one  in  which  operational  equipment  and  actual  terrain  is  used, 
such  as  when  a  platoon  maneuvers  in  its  tanks.  Virtual  exercises 
involve  the  networking  of  simulators  to  make  it  possible  for 
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crews  to  interact  together  on  a  common  terrain  database. 
Information  produced  by  each  simulator,  such  as  its  location  on 
the  terrain  database,  is  transmitted  over  a  network  and  picked  up 
by  other  simulators.  The  graphics  generator  for  each  simulator 
employs  network  data  and  data  from  a  common  terrain  database  to 
provide  a  current  "out  the  window"  view  of  the  world  for  crew 
members  (Thorpe,  1987) .  Constructive  simulations  represent 
military  units  as  an  aggregate  without  simulating  each  entity 
within  a  unit  (Stober,  Kraus,  Foss,  Franceschini,  and  Petty, 

1995) ,  and  this  environment  has  been  used  largely  to  support 
command  and  staff  training.  A  mixed  or  synthetic  theater  of  war 
(STOW)  environment  contains  a  mix  of  live,  virtual,  or 
constructive  envirohments  (Sottilare,  1995) . 


Purpose  of  Report. 

This  report  describes  the  factors  influencing  the  development 
of  materiel  and  behavioral  systems  to  support  AARS  in  live, 
virtual,  and  constructive  environments.  This  information  is 
relevant  to  those  responsible  for  defining  the  requirements  for 
AAR  systems,  and  it  is  relevant  to  research,  development,  and 
engineering  (RD&E)  organizations  responsible  for  expanding  the 
technology  base  to  meet  future  training  challenges. 

Background 

Figure  1  provides  an  overview  of  the  process  of  developing 
AAR  systems  as  it  existed  in  the  1990  timeframe.  This  date 
is  important  because  this  was  when  AAR  system  users  and  the  U.S. 
Army  Simulation,  Training  and  Instrumentation  Command  (STRICOM) 
were  defining  the  requirements  for  the  Close  Combat  Tactical 
Trainer  (CCTT)  AAR  system.  A  description  of  user  (e.g.,  the 
Armor  school)  requirements  initiates  the  development  process.  To 
a  large  extent  these  requirements  are  based  upon  the  users 
familiarity  with  the  device  being  replaced.  The  next  step  in  the 
process  involves  increased  specification  of  requirements  by^ 
looking  at  the  technology  base  to  examine  current  capabilities 
and  technical  issues  relevant  to  the  system  under  development 
(Meliza  &  Lampton,  1991)  .  ’While  examining  the  technology  base, 
STRICOM  engineers  and  other  members  of  the  government  RD&E 
community  provided  feedback  regarding  the  direction  of  future 
growth  in  the  technology  base.  The  specifications  for  most 
training  devices  automatically  include  those  developed  under  the 
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ongoing'  series  of  workshops  on  the  Standards  for  the 
Interoperability  of  Defense  Simulations  (Institute  for  Electrical 
and  Electronics  Engineers,  1993) ,  frequently  referred  to  as  the 
Distributed  Interactive  Simulation  (DIS)  Standards. 


Figure  1,  The  AAR  system  development  process  Circa  1990 

There  were  a  number  of  problems  with  the  process  shown  in 
Figure  1.  First,  users  had  very  little  experience  with  platoon 
and  company  level  AAR  systems  to  draw  upon.  Second,  the 
technology  base  for  AAR  systems  was  in  its  early  stages.  Third, 
the  DIS  Standards  did  not  reflect  unit  performance  measurement 
and  AAR  system  issues  due  to  labk  of  experience.  Fourth,  the  AAR 
system  is  only  one  component  of  the  training  system,  and  the 
requirements  for  an  AAR  system  must  be  developed  in  coordination 
with  other  training  components. 

Simulation  Networking  (SIMNET) ,  the  networked  simulation 
system  to  be  replaced  by  CCTT  was  intentionally  developed  without 
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a  performance  measurement  system  (Alluisi,  1991) .  The  first 
tools  developed  within  the  SIMNET  program  to  support  AARs  were 
limited  to  animated  replays  from  a  plan  view  perspective  or  "out 
the  window"  perspective .  The  number  of  systems  available  to 
support  AARs  were  often  fewer  than  the  number  of  exercises 
conducted  concurrently  at  a  training  site,  meaning  that  most  AARs 
were  conducted  without  the  benefit  of  data  displays  (Goldberg  and 
Meliza,  1993) . 

Attempts  to  implement  systems  with  added  measurement 
capabilities  were  frustrated  by  periodic  changes  in  the  SIMNET 
data  stream,  requiring  software  changes  in  the  measurement 
system.  For  example,  the  Unit  Performance  Assessment  System 
(UPAS)  was  developed  to  support  measurement  with  SIMNET  Version 
5.0  just  as  this  version  was  replaced  by  6.0  (Meliza,  Bessemer, 

&  Tan,  1994) .  A  version  of  the  UPAS  capable  of  using  SIMNET  6.0 
data  was  ready  for  testing  just  as  SIMNET  Version  6.6.1  replaced 
6.0.  Fortunately,  6.6.1  was  the  last  version  of  SIMNET.  The 
upgi'ad.e  of  the  UPAS  to  SIMNET  6.6.1  allowed  testing  of  the 
software  in  its  application  to  AARs. 

By  1993,  we  were  collecting  lessons  learned  from  various 
SIMNET  AAR  systems  and  performance  measurement  systems  that  could 
provide  input  for  refining  the  requirements  for  CCTT  AAR  systems 
as  well  as  requirements  for  AAR  systems  for  other  training 
devices.  A  new  problem  emerging  was  the  duplication  of  efforts 
to  develop  AAR  systems.  The  Project  Manager  for  Combined  Arms 
Tactical  Trainer  (PM-CATT)  asked  ARI  to  organize  an  Army  AAR 
Conference  for  early  1994  in  an  effort  to  reduce  redundancy  in 
efforts  to  develop  AAR  systems  and  to  help  formalize  and 
standardize  Army  requirements  for  AAR  systems.  An  important  part 
of  this  conference  was  the  demonstration  of  new  and  developing 
AAR  capabilities  and  lessons  learned.  This  conference  included 
the  live,  virtual  and  constructive  training  environments  under 
the  assumption  that  they  have  common  AAR  requirements .  The 
presentations,  discussions,  and  conclusions  from  this  conference 
were  documented  and  distributed’ by  ARI  (1994)  . 

I 

One  important  finding  from  this  conference  was  that  AAR 
system  requirements  cannot  be  defined  in  isolation  from  other 
components  of  training,  such  as  the  exercise  management  system. 
Sponsorship  for  the  second  AAR  Conference,  held  in  April  of  1995 
and  again  organized  by  ARI,  increased  to  include  the  U.S.  Army 
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Training  and  Doctrine  Command  (TRADOC)  Assistant  Deputy  Chief  of 
Staff  for  Training  and  the  Commanding  General  of  STRICOM.  A 
major  benefit  of  this  new  alliance  is  that  it  brought  the  AAR 
Conference  into  the  Warfighter  XXI  program  under  which  the  AAR 
system  is  one  of  five  integrated  training  components  (Marlin, 

1995) ,  The  first  is  the  Standard  Army  Training  System  (SATS) 
component  with  the  mission  of  automating  training  management  to 
make  more  effective  use  of  training  resources.  The  second 
component  is  Training  Support  Packages  (TSPs)  with  the  mission  of 
providing  an  automated,  structured  situational  training  template 
resourced  to  generate  training  events .  This  component  will 
provide  scenarios  for  use  with  specific  training  aids,  devices, 
simulations,  and  simulators  (TADDS) .  The  third  component  is  the 
TADDS  with  the  mission  of  providing  integrated  and  effective 
training  tools  to  efficiently  train  a  unit .  The  fourth  component 
is  the  Standard  Army  AAR  System  (STAARS)  with  the  mission  of 
providing  a  standardized,  automated  storage  and  distribution 
system  to  support  training  evaluation,  resource  utilization  and 
data  analysis  for  lessons  learned  efforts.  The  fifth  component 
is  the  Army  Training  Digital  Library  (ATDL)  with  the  broad 
mission  of  providing  access  to  information  to  assist  Army 
trainers . 

The  Warfighter  XXI  strategy  is  intended  to  address  the  total 
Army,  employ  institutional  and  self -development  strategies 
tailored  to  support  collective  training,  be  supportable  with 
limited  resources,  and  make  use  of  future  technologies  and 
capabilities.  The  last  two  requirements  impose  two  critical 
certain  restrictions  on  how  the  Army  must  develop  future  training 
systems;  the  Army  must  avoid  redundant  development  and 
experimentation,  and  the  Army  must  not  adopt  technological 
solutions  to  problems  too  soon. 

The  presentations  and  findings  of  the  Second  AAR  Conference 
were  summarized  by  the  ARI  Simulator  Systems  Research  Unit 
(1995)  .  The  addition  of  Warfighter  XXI,  STAARS,  and  AAR 
Conferences  has  many  effects  up6n  the  AAR  system  development 
process  as  illustrated  in  figure  2 .  The  development  and 
application  of  AAR  systems ‘has  expanded  the  technology  base, 
provided  users  with  examples  of  current  capabilities,  identified 
problems  to  be  addressed  by  the  technology  base,  and  provided 
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input  to  the  DIS  standards.  In  addition,  these  research  and 
development  efforts  have  provided  input  to  Warfighter  XXI  and  the 
development  of  STAARS . 


Figure  2.  Effects  of  Warfighter  XXI,  the  STAARS,  and  AAR 
conferences  on  the  AAR  system  development  process 


The  core  capabilities  of  a  STAARS  were  briefly  described  in 
an  iterative  fashion  culminating  in  a  Nov  95  STAARS  Mission  Needs 
Statement  (MNS) .  According  to  the  MNS  document,  STAARS  must: 


o  be  compatible  and  interoperable  in  the  constructive, 
virtual,  and  live  environments  to  include  joint  and 
combined  systems ; 

l 

o  compare  unit  performance  to  doctrinal  standards  and  lessons 
learned  from  other  training  events; 

o  provide  high  quality  standardized  AAR  products  appropriate 
to  the  echelon (s)  being  trained  or  analyzed; 
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o  retrieve,  display,  and  distribute  feedback  without 
disrupting  the  TADSS,  test,  experiment,  or  training 
exercise; 

o  standardize  the  automated  storage,  distribution,  and 
retrieval  of  AAR  data  within  the  ATDL  architecture; 

o  provide  for  identification  of  specific  events  by  using 
tailored  parameters  to  alert  the  user;  and 

o  provide  standardized  user  definable  products  incorporating 
playback  capability,  C4I  (command,  control,  communication, 
computers,  anid  intelligence) /video  products,  access  to 
doctrinal  sources,  statistical  products,  terrain  analysis, 
and  trainer  observations. 

An  initial  STAARS  action  plan  has  been  developed  to  provide 
milestones  and  responsibilities  for  the  iterative  development  of 
STAARS.  This  plan  assumes  iterative  development  of  the  STAARS 
over  the  next  ten  years.  A  key  milestone  is  the  development  of  a 
STAARS  Operational  Requirements  Document  (ORD)  in  FY96  with  the 
National  Simulation  Center  (NSC)  serving  as  the  lead  agency. 

The  goal  of  the  third  AAR  Conference,  held  in  Jan  96,  was  to 
revise  a  draft  ORD  and  expand  upon  the  STAARS  Action  Plan.  The 
remainder  of  this  report  provides  input  to  the  ORD  and  Action 
Plan. 

The  series  of  DIS  Standards  Conferences  that  began  in  Aug  89 
are  concerned  with  describing  what  a  simulation  system  must  do  to 
interoperate  with  other  systems  (Institute  for  Simulation  and 
Training,  1992)  .  The  initial  output  from  the  DIS  Standards 
Conferences  relevant  to  the  development  of  AAR  systems  was 
limited  to  a  standard  (Institute  of  Electrical  and  Electronic 
Engineers,  [IEEE]  1993)  providing  descriptions  of  the  network 
data  stream  or  protocol  data  units  (PDUs)  used  to  support  the 
communication  of  information  ambng  entities.  The  input  from  the 
Conference  has  expanded  tOj include  a  draft  document  describing 
recommended  practices  for  exercise  management  and  feedback  in  the 
DIS  environment  (Institute  for  Simulation  and  Training,  1995)  and 
a  document  presenting  the  rationale  behind  the  recommended 
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practices  (Institute  for  Simulation  and  Training,  1994)  .  The 
present  report  describes  relationships  between  DIS  standards  and 
the  STAARS  concept . 
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Designing  AAR  Aids  to  Help  Document  What  Happened, 

Explain  What  Happened. and  Identify  Potential  Corrective  Actions 


Background 

AAR  "systems"  should  support  the  goals  of  analyzing  what 
happened  during  an  exercise,  deciding  why  it  happened,  and 
identifying  potential  corrective  actions.  Analysis  of  outcome 
data  must  go  beyond  battle  damage  assessment  (e.g.,  counting  the 
number  of  weapon  systems  on  each  side  that  are  damaged  or 
destroyed)  to  include  examination  of  key  mission  subtasks  by 
answering  such  questions  as :  was  the  unit  slow  to  respond  to 
enemy  contact?  Didj the  unit  apply  an  adequate  volume  and  mix  of 
fires  against  the  enemy?  Were  fires  distributed  appropriately 
among  specific  targets  or  areas? 

At  many  points  during  the  AAR  the  "data"  used  to  examine  the 
exercise  shifts  from  external  representations  of  ground  truth  to 
the  knowledge  and  memories  of  exercise  participants.  This  shift 
is  often  essential  to  the  goal  of  diagnosing  performance  problems 
to  a  point  that  supports  the  identification  of  specific 
corrective  actions.  In  looking  through  the  list  of  performance 
problems  in  Table  1,  note  that  none  could  be  identified  without 
information  from  participants  about  their  perceptions  of  events 
and  knowledge  of  tactics,  techniques,  and  procedures  (TTPs) . 
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Table  1. 


List  of  Representative  Performance  Problems  that  might  be 
Identified  during  an  AAR _ _ _ 

o  Unit  members  have  not  learned  existing  standard  operating 
procedures  (SOPs)  or  are  confused  about  SOP  interpretation 

o  Important  information  in  general  (or  of  a  specific  type)  is 
not  being  disseminated  up  or  down  the  chain-of -command 

o  The  SOP  of  one  unit  is  not  compatible  with  that  of  another 

o  There  is  confusion  about  the  purpose  and  scope  of  a  unit 
task  or  about  how  to  apply  a  particular  tactical 
principle 

o  Subordinate  leaders  have  a  difficult  time  analyzing  the 
tactical  situation  or  applying  the  results  of  their 
analyses . 


STAARS  and  PIS  Standards  Guidance 

The  MNS  for  the  STAARS  contains  three  capabilities  statements 
that  relate  to  the  design  and  content  of  AAR  aids.  First,  the 
system  must  provide  the  capability  to  compare  unit  performance  to 
doctrinal  standards  and  lessons  learned  from  other  training 
events.  Second,  the  system  should  be  compatible  and  inter¬ 
operable  in  the  constructive,  live  and  virtual  environments,  and 
it  should  provide  high  quality  standardized  AAR  products 
appropriate  to  the  echelons  being  trained  or  analyzed.  Third, 
STAARS  will  provide  standardized  user  definable  products 
incorporating  playback  capability,  C4I/video  products,  access  to 
doctrinal  resources,  statistical  products,  terrain  analysis,  and 
observer/controller  (0/C)  observations. 

I 

The  recommended  practices  for  exercise  management  and 
feedback  in  the  DIS  environment  address  data  collection, 
presentation,  and  analytic  functions  (Institute  for  Simulation 
and  Training,  1995) .  Many  of  these  recommended  practices  concern 
the  PDU  data  stream  used  to  communicate  information  among 
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entities  in  a  networked  exercise,  such  as  the  Entity  Appearance 
PDU  stream  generated  by  each  entity  to  convey  information  about 
its  status  to  other  entities.  According  to  the  recommended 
practices  document,  a  DIS  feedback  system  should: 

o  reproduce  the  PDU  stream  exactly  as  received; 

o  archive  non- PDU  data  important  in  examining  unit 
performance  and  index  to  the  PDU  data; 

o  archive  the  PDU  and  non -PDU  data  in  a  manner  that  supports 
analysis  within  and  across  exercises; 

o  allow  user  to'  select  PDUs  to  be  collected  from  the  network; 

o  support  selectable  checks  to  validate  events  determined  by 
two  or  more  PDUs  (e.g.,  a  detonation  PDU  should  be 
associated  with  each  fire  PDU) ; 

o  provide  an  out-the-window  view  of  the  exercise  with  a 
viewpoint  selectable  by  the  user; 

o  provide  a  plan  view  of  the  exercise  that  depicts  entities, 
topographic  features,  cultural  features,  and  user-defined 
annotations  (e.g.,  control  measures)  and  allow  user  to 
select  what  is  to  be  depicted; 

o  pan,  zoom,  and  adjust  map  display  scale; 

o  control  playback  with  fast  f orward/reverse ,  jump 

forward/reverse  (move  forward  or  back  to  specific  points  in 
time  without  replaying  all  of  the  interpolated  data) , 
selection  of  play  speeds,  single  frame  movement,  and 
pause/freeze  capability; 

o  support  playback  audio; 

o  provide  hardcopy  outputs  of  displays; 

o  allow  the  user  to  select  specific  entities  or  classes  of 
entities  for  replay; 

o  show  environmental  effects; 
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o  show  movement  of  articulated  parts; 

o  support  overlay  of  non-PDU  data  (e.g.,  control  measures) ; 

o  replay  exercises  at  a  level  appropriate  to  the  exercise 
participants  being  debriefed  against  appropriate 
environmental  models  using  changes  in  entity  icons  and 
other  graphical  aids  to  indicate  all  relevant  variables  and 
status  changes; 

o  display  data  on  line -of -sight  among  entities  on  command; 

o  support  user  editing  of  displays;  and 

o  support  user  customization  of  tabular  displays  and  figures 
including  graphs,  tables,  summaries  and  time  lines  showing 
critical  exercise  events  as  defined  by  the  user. 

History  and  Status  of  Research 

Historical  Perspective.  The  development  of  data  displays  to 
support  AARs  is  a  young  endeavor.  The  move  towards  tactical 
engagement  simulation  (TES)  with  Squad  Combat  Operations 
Engagement  Simulation  (SCOPES)  and  realistic  training  (REALTRAIN) 
in  the  seventies  might  be  considered  a  start  point  for  the 
development  of  AAR  aids  by  providing  engagement  outcomes  that 
were  more  objective  than  the  opinions  of  field  umpires  (Anderson 
&  Sherwood,  1975)  .  The  subsequent  adoption  of  the  Multiple 
Integrated  Laser  Engagement  System  JMILES)  in  live  exercises  as 
the  approach  for  simulating  weapons  effects  further  enhanced  the 
objectivity  of  mission  outcomes  (Loftis,  1980;  Sulzen,  1986) . 

The  data  displays  used  for  AARs  were  generally  limited  to  those 
reflecting  casualty  exchange  ratios. 

The  subsequent  move  towards  instrumented  ranges  at  the 
National  Training  Center  (NTC)  in  the  early  eighties  further 
increased  the  data  available  to*  describe  what  happened  during 
exercises.  These  data  included  TES  combined  with  position 
location  data  that  could  b6  displayed  over  plan  view  maps,  and  it 
included  videotapes  of  the  battlefield  and  recordings  of  tactical 
communications.  With  few  expectations,  the  data  displays  at  the 
NTC  focused  on  battalion  task  level  performance  rather  than 
company  or  platoon-level.  Further,  there  were  problems  with  the 
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quality  of  the  TES  data  resulting  in  a  situation  where  only  a 
small  percentage  of  casualties  could  be  linked  to  specific  firing 
events  (Walsh  &  Keesling,  1993) . 

The  addition  of  networked  simulators  to  create  an  electronic 
battlefield  in  the  mid  to  late  eighties  greatly  increased  the 
data  that  might  be  used  for  preparing  AAR  aids.  For  example,  we 
went  from  a  situation  where  vehicle  status  data  were  limited  to 
periodic  updates  regarding  the  location  and  damage  status  of 
vehicles  to  the  point  of  having  near  continuous  updates  on  the 
locations  of  vehicles,  the  speed  at  which  vehicles  were  moving, 
the  orientation  of  vehicles,  engine  speeds,  the  orientation  of 
gun  tubes,  the  elevation  of  gun  tubes,  fuel  levels,  and 
ammunition  levels  (Meliza,  Bessemer,  and  Tan,  1994)  .  However,  as 
mentioned  on  page  four  of  this  report,  efforts  to  develop  data 
displays  using  these  new  sources  of  data  for  training  feedback 
displays  did  not  meet  with  much  success  until  the  early  nineties. 

The  initial  SIMNET  did  not  have  the  capability  to  provide 
data  displays  that  could  be  used  to  assess  performance  (Alluisi, 
1991) .  In  the  past  five  years  we  have  learned  important  lessons 
about  the  design  of  data  displays  and  about  the  capability  of 
networks  to  provide  the  data  we  need  to  create  effective 
displays . 

TTsina  AAT?  Aids  to  Show  Ground  Truth.  AAR  aids  can  play  the 
important  role  of  documenting  what  happened  during  an  exercise. 
One  of  the  major  benefits  of  moving  towards  an  electronic 
battlefield  is  that  it  makes  it  easier  to  document  what  happened 
during  an  exercise.  Instead  of  a  trainer  telling  a  unit  that  it 
failed  to  provide  adequate  returning  fire  when  engaged  by  the 
enemy,  the  trainer  can  replay  a  portion  of  the  exercise  showing 
slow  and  meager  unit  fires  against  a  heavy  volume  of  enemy  fires 
or  provide  a  graph  summarizing  the  same  information.  Exercise 
participants  can  then  see  for  themselves  what  happened  and  reach 
their  own  conclusions  about  how  well  they  employed  direct  fire 
against  the  enemy. 

Comparina  Behavior  with  Doctrinal  Standard?  ^hd  Lesgons. 
T.parnad.  The  purpose  of  collective  performance  measurement  is  to 
find  out  how  well  individuals  work  together  in  performing  group 
tasks.  For  Army  organizations,  these  tasks  are  defined  in 
Mission  Training  Plan  (MTP)  documents  (U.S.  Army  Training  and 
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Doctrine  Command,  1984),  such  as  the  MTP  for  the  Tank  Platoon 
(Department  of  the  Army,  1988) .  MTP  documents  include  standards 
for  evaluating  how  well  a  unit  performs  each  task  and  subordinate 
subtasks.  For  example,  the  overall  standard  for  the  task  of 
"execute  a  wedge  formation"  is  "the  platoon  executes  the  wedge 
formation  without  delay  and  without  stopping  movement".  There 
are  also  subtask  standards  that  might  be  viewed  as  descriptions 
of  task  steps.  One  standard  for  the  subtask  "the  platoon 
executes  the  wedge  formation  without  delay"  is  "Pltldr  positions 
himself  at  either  the  1  o'clock  or  11  o'clock  position^^where  he 
can  best  control  his  platoon  and  according  to  his  SOP. 

In  addition  to  comparing  unit  performance  with  performances 
described  in  doctrinal  standards,  an  AAR  system  should  also  help 
a  trainer  compare  the  behavior  of  a  unit  with  descriptions  of  the 
behavior  of  successful  and  unsuccessful  units.  For  example,  the 
Center  for  Army  Lessons  Learned  (CALL)  has  examined  performances 
of  units  at  the  NTC  and  described  actions  typical  of  well  trained 
or  inadequately  prepared  units.  As  part  of  this  larger  effort. 
Goldsmith  and  Hodges  (1987)  prepared  a  report  containing 
recommendations  for  improving  tactical  reconnaissance  at  the 
battalion  level,  based  upon  lessons  learned  at  the  NTC.  Of 
course  the  Army  wants  to  provide  units  with  displays  that  can  be 
used  to  quickly  see  what  other  successful  units  do  in  a  specific 
situation,  rather  than  providing  the  unit  with  a  report  to 
and  analyze.  This  approach  fits  the  "A  way"  concept  recommended 
by  LTG  (Ret)  Brown  whereby  units  are  shown  a  display  that  shows 
one  way  of  doing  something  that  led  to  success,  without  implying 
that  it  is  the  only  way  to  be  successful. 

The  displays  that  might  be  used  to  compare  the  performance  of 
a  unit  with  standards  or  lessons  learned  are  not  limited  to  those 
that  illustrate  the  outcomes  of  mission  and  task  performance. 

They  might  include,  for  example,  a  list  of  innovative  command  and 
control  procedures  used  by  units  at  the  NTC  to  make  more 
efficient  use  of  dozers  when  preparing  for  defensive  operations. 
The  items  on  this  list  might  be* compared  with  what  unit  members 
remember  doing  to  support  efficient  dozer  use  during  a  CCTT 
exercise  to  see  if  there  are  new  ideas  on  the  list  from  the  NTC. 
That  is,  it  may  not  be  necessary  to  have  a  display  based  on  the 
activities  of  the  unit  in  CCTT. 
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Importance  of  Being  Able  to  Rapidly  Intgypygt  th^ 

Information  Contained  in  AAR  Aids.  An  important  component  of  the 
STAARS  concept  is  the  standardization  of  AAR  aids  for  each 
echelon  across  training  environments.  If  a  tank  platoon  is  using 
SIMNET  to  conduct  training  in  preparation  for  training  at  the 
NTC,  then  the  aids  used  for  SIMNET  exercises  should  be  the  same 
as  those  used  for  tank  platoons  at  the  NTC.  A  major  benefit  of 
standardizing  displays  across  environments  is  that  it  links 
exercises  with  capstone  training  environments,  such  as  rotations 
to  a  combat  training  center  (CTC) .  For  example,  if  a  CTC 
employed  a  particular  data  table  to  show  how  well  a  unit  kept  its 
vehicles  ready  for  combat,  units  would  want  to  see  the  same  table 
after  participating!  in  virtual  exercises. 

It  is  important  that  the  displays  used  be  immediately 
interpretable  to  exercise  participants.  If  many  minutes  are 
required  to  explain  what  a  display  means  then  it  will  be  of 
little  or  no  value  in  the  training  environment.  Standardizing  of 
displays  within  echelons  and  across  training  environments  helps 
to  make  sure  that  exercise  participants  will  be  familiar  with  the 
displays  presented  in  AAR  sessions. 

TTsina  AAR  Aids  to  Support  the  Trainer's  Analvgis  of 
Performance .  Data  displays  play  at  least  three  different  roles 
in  the  AAR  process  (Meliza,  Bessemer,  Burnside,  &  Shlechter, 

1992) .  First,  the  trainer  may  use  aids  to  illustrate  the 
strengths  and  weaknesses  to  a  unit.  Second,  they  can  be  used  to 
suggest  alternative  courses  of  action  to  a  unit.  Third,  a 
trainer  may  use  displays  to  diagnose  the  strengths  and  weaknesses 
of  a  unit  in  preparation  for  conducting  the  AAR.  It  is  crucial 
that  aids  used  for  the  first  two  purposes  be  immediately 
interpretable  to  all  exercise  participants,  but  aids  used  for  the 
third  purpose  do  not  need  to  meet  this  requirement.  However,  we 
do  want  to  make  sure  that  very  little  time  is  required  to  prepare 
users  to  employ  AAR  aids  in  diagnosing  unit  training  needs. 


Measures  of  the  degree  of  dispersion  of  vehicles  within  a 
battalion  task  force  have  been  shown  to  be  correlated  with 
mission  outcomes  at  the  Arttiy's  NTC  (Goehring  &  Sulzen,  1994)  . 

Too  much  time  might  be  required  to  explain  these  data  displays 
and  statistics  for  them  to  be  employed  during  an  AAR,  but  the 
analyses  can  be  used  to  assess  command  and  control  strenghths  and 
weaknesses  on  which  a  trainer  might  want  to  focus  during  the  AAR. 
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To  use  such  displays  for  diagnostic  purposes,  the  program  of ^ 
instruction  (POI)  for  the  system  user  must  incorporate  training 
on  how  to  use  these  displays  or  the  system  should  apply  these 
displays  in  a  manner  that  is  transparent  to  the  user  (e.g., 
provide  the  user  with  the  interpretation  of  the  display  rather 
than  showing  the  display  to  the  user) . 

Substantial  research  and  development  activities  are  required 
to  develop  data  displays  that  can  be  used  to  measure  unit 
performance  and/or  diagnose  unit  strengths  and  weaknesses. 

Once  such  diagnostic  tools  have  been  developed  there  may  often  be 
a  need  to  design  new  displays  that  help  to  illustrate  strengths 
and  weaknesses  during  the  AAR.  For  example,  in  the  case  of  the 
measures  of  dispersion  related  to  mission  outcomes,  we  need  ways 
to  let  a  leader  see  that  his  forces  are  too  spread  out  to  support 
effective  execution  of  mission  tasks. 

The  Need  for  New  AAR  Aids.  Conduct  of  training  in  an 
electronic  battlespace  supports  preparation  of  AAR  aids  by 
providing  data  in  an  electronic  format,  but  network  data  alone 
are  not  sufficient  to  measure  unit  performance.  Consider  the 
standard  "platoon  occupies  position  designated  in  operations 
order  and  moves  to  turret-down  positions"  from  the  task  "perform 
consolidation  and  reorganization  activities"  from  the  MTP  for  the 
Tank  Platoon  (Department  of  the  Army,  1988)  .  To  apply  this 
standard  one  must  know  the  locations  of  vehicles  (from  network 
data) ,  the  position  designated  in  the  operations  order  (from 
planning  data) ,  and  the  terrain  situation  (from  the  terrain 
database) .  Deciding  how  well  a  unit  performed  with  respect  to 
MTP  standards  often  requires  a  mix  of  network  data,  data  on  radio 
communications,  planning  data,  terrain  data,  and  direct 
observations  of  human  behavior  (Meliza,  1993b) .  Further, 
complicating  the  process  of  applying  these  diverse  data  sources 
is  the  fact  that  these  data  must  also  be  considered  as  a  function 
of  time. 

Innovative  design  work  is  nfeeded  to  create  displays  that 
bring  together  data  from  mixed  sources  over  time.  One  example  of 
such  an  aid  is  the  UPAS  Fitefight  Display,  designed  by  a  trainer 
who  saw  the  need  for  a  display  to  help  decide  where  and  when  a 
unit  concentrates  direct  and  supporting  fires  (Meliza,  Tan,  & 
Bessemer,  1994) .  A  Firefight  Display  (Figure  3)  shows  direct  and 
indirect  firing  events  over  a  terrain  map,  covering  a  user 
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selectable  period.  Direct  firing  events  are  displayed  with  shot 
lines  connecting  the  location  of  the  firing  vehicle  with  that  of 
the  vehicle  or  ground  impact,  and  a  vehicle  icon  is  used  to  show 
the  location  of  the  firing  vehicle  using  the  same  color  coding 
system  as  the  UPAS  Plan  View.  A  miss  is  indicated  by  a  white 
line,  and  a  green  line  indicates  a  hit  or  a  catastrophic  kill. 

If  the  firing  event  results  in  a  kill,  there  will  also  be  a  dead 
vehicle  icon  at  the  target  location  (cyan  for  a  destroyed  BLUFOR 
vehicle  and  white  for  a  destroyed  REDFOR  vehicle) .  Artillery 
impacts  are  shown  using  white  rectangles. 


Fi^Kt  Start  Tina:  150500 

Fire  Tine:  151200 
End  Tine;  151200 


Figure  3 .  Example  of  a  fire  fight  display 

The  UPAS  Exercise  Timeline  is  another  example  of  an  AAR  aid 
that  integrates  a  mix  of  data  over  time.  The  Exercise  Timeline 
describes  firing,  movement,  and  communication  events  as  a 
function  of  time  and  control  measures  to  help  make  it  more 
obvious  how  different  kinds  of  fevents  are  related  in  time. 

The  Exercise  Timeline  can  be  used  to  identify  points  in  time  that 
warrant  examination  using  other  data  displays,  and  it  can  be  used 
by  itself  to  assess  unit  performance.  For  example,  one  might  use 
the  Exercise  Timeline  to  find  out  if  a  platoon  in  the  offense 
halted  soon  after  being  engaged  by  (or  engaging)  the  enemy  and 
promptly  reported  contact . 
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Figuirs  4  provid0S  an  example  of  an  Exercise  Timeline .  The 
top  line  covers  the  time  between  the  start  and  end  of  the 
exercise,  but  the  user  can  change  the  display  to  focus  on  a 
smaller  span  of  time.  The  second  line  describes  platoon  movement 
as  a  function  of  time  and  unit  control  measures.  The  bars  at  the 
bottom  of  this  line  indicate  when  the  first  and  last  vehicle  of  a 
unit  crossed  a  control  measure.  In  the  example,  LD  Tin  and 
Assembly  Area  Gold  were  right  next  to  each  other,  so  the  times 
for  vehicle  crossing  are  identical.  The  unit  crossed  Phase  Line 
Silver  quickly  but  crossed  Phase  Line  Bronze  as  sections  with  the 
rear  element  overwatching  fire  and  maneuver  of  the  lead  element. 


EXERCISE  IDr 

- SImnCT  EXLkClSE  TIMlLinE 

001  DATE;  B/ 12/92 

COMPANY: 

C 

14:40 

PI  T  1 

50  15:00  15:10  15:20 

LD  TIN  BRONZE 

GOLD  SILVER 

15:30 

CPS 

15:40 

8^e 

MOVE 

1 

' 

SHOOT - 

{  o 

O 

R  R  R 

O  ?  R  O  O 

F 

— 

Move  Legend: 

■  Time  between  first  end  last  vehicle  crossing  the  control  measure. 

^  »  Time  during  which  no  vehicle  moved. 

Shoot  Legend:  ^  First  friendly  fire  delivered, 

o  Artillery  fire  near  unit.  X  Enemy  vehicle  destroyed, 

f  First  enemy  fire  received.  O  Friendiy  vehicle  destroyed. 


Comm  Legend: 

R  Report 
F  Call  for  ffre 


M  Miscellaneous 
0  Orders 

?  Request  for  Information 


Figure  4.  Example  of  an  exercise  timeline 

Disabled  or  destroyed  vehicles  are  not  counted  in  computing  when 
control  measures  are  crossed.  The  Exercise  Timeline  also 
indicates  the  beginning  and  ending  of  periods  when  the  entire 
platoon  was  halted. 

The  third  line  provides  information  about  the  time  of  direct 
and  indirect  fires.  A  small  square  is  used  to  indicate  the 
platoon  received  artillery  fire,  an  arrow  pointed  down  indicates 
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ths  first  cnsmy  dirsct  firs  was  rsceived.  h>y  ths  platoon,  an  arrow 
pointed  up  indicates  the  first  platoon  fires  on  the  enemy,  a 
small  "X"  indicates  when  an  enemy  vehicle  is  destroyed,  and  a 
small  circle  indicates  when  a  friendly  vehicle  is  destroyed. 

The  fourth  line  provides  information  about  the  timing  and 
type  of  communications  over  the  tactical  radio  network.  The 
communication  line  of  the  Exercise  Timeline  for  each  platoon  and 
the  company  commander  indicates  when  each  of  five  types  of 
tactical  communications  are  transmitted  by  the  platoon  or  company 
commander  over  the  company  net .  An  order  is  indicated  by  an  0  , 
a  report  is  indicated  by  an  "R",  a  call  for  fire  is  indicated  by 
an  "F",  and  a  request  for  information  is  indicated  by  a  question 
mark.  Tactical  communications  that  do  not  fall  into  one  of  these 
four  types  are  classified  as  miscellaneous  and  indicated  with  an 
"M"  on  the  Exercise  Timeline. 

An  example  of  new  type  of  display  developed  for  use  in  the 
constructive  environment  is  a  version  of  an  overhead  snapshot  of 
the  battlefield  showing  the  density  of  coverage  of  various  areas 
of  the  battlefield  by  friendly  weapons  (Fernan  &  Dryer,  1994) . 
This  figure  is  created  by  assessing  the  line-of-sight  fans  from 
each  friendly  system  and  considering  the  number  of  systems  haying 
line-of-sight  with  each  point.  An  example  of  an  innovative  live 
display  is  that  used  by  Goehring  and  Sulzen  (1994)  to  assess 
dispersion  of  forces  at  the  NTC. 

There  is  still  a  need  for  new  types  of  innovative  displays, 
because  unit  performance  measurement  is  a  complex  challenge .  In 
many  cases,  a  half  dozen  or  so  data  displays  might  be  required  to 
thoroughly  examine  an  aspect  of  unit  performance,  due  to  the  fact 
that  unit  actions  should  reflect  the  overall  mission,  enemy, 
terrain  (and  weather) ,  troops,  and  time  available  (METT-T) 
situation.  For  example,  a  unit  might  fail  to  return  fire  at  a 
particular  point,  because: 

o  the  enemy  is  out  of  range; 

i 

o  fires  are  masked  by  another  friendly  element; 

o  the  unit  is  not  aware  it  is  being  fired  upon; 

o  line-of-sight  is  blocked  by  terrain  features; 
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o  the  unit  believes  it  has  not  actually  been  sighted  by  the 
enemy  and  will  give  away  its  position  if  it  fires; 


o  unit  SOP  specifies  that  fires  are  to  be  withheld  until  the 
leader  gives  a  fire  command  but  the  leader  is  dead  or  has 
communication  problems;  and 

o  the  unit  is  out  of  ammunition,  has  suffered  catastrophic 
kills  and  firepower  kills  from  previous  engagements. 

New  types  of  data  displays  might  integrate  information  across 
existing  data  displays  or  otherwise  reduce  the  number  of 
potential  displays  a  trainer  might  need  to  employ.  Unfortunately 
there  is  no  source  one  can  use  to  identify  all  of  the  standards 
for  which  we  need  improved  data  displays.  However,  there  is  an 
online  database  that  identifies  the  data  sources  (network, 
terrain  data,  planning  data,  communications  data,  and  direct 
observation  of  the  behavior  of  humans)  needed  to  apply  each  of 
approximately  5000  MTP  standards  at  the  armor  platoon,  company 
team,  and  battalion  task  force  levels  (Meliza,  1993b) . 

Hixson  (1996) ,  in  a  description  of  AAR  requirements  for 
exercises  at  brigade  level  and  higher  supported  by  the  Corps 
Battle  Simulation  (CBS)  AAR  system,  brought  up  the  important 
point  that  more  of  the  critical  feedback  for  these  exercises  is 
based  upon  direct  observation  of  humans  and  less  is  based  upon 
electronic  data.  To  obtain  these  data  requires  the  development 
of  a  detailed  data  collection  and  analysis  plan  that  integrates 
data  collected  across  observers.  Integration  is  required  to,  for 
example,  provide  information  that  can  be  used  to  assess  how  well 
the  maneuver  planning  and  monitoring  process  is  synchronized  with 
the  logistical  planning  and  monitoring  process. 

Measuring  Generic  Team  Skills.  Another  approach  to  assessing 
unit  performance  and  identifying  potential  corrective  actions 
might  focus  on  team  behaviors  that  are  not  task  specific,  such  as 
examining  patterns  of  communication  (Blickensderf er,  Cannon- 
Bowers,  &  Salas,  1994;  Urban,  Bowers,  Monday,  &  Morgan,  1993). 
Urban  et  al.,  for  example, ‘ showed  that  poor  performing  teams  used 
a  question  and  answer  sequence  to  communicate  information,  while 
higher  performing  teams  were  less  likely  to  use  questions  to 
prompt  the  dissemination  of  information.  That  is,  members  of 
high  performing  teams  appear  to  anticipate  the  information  needs 
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of  other  team  members  and  provide  this  information  without  being 
asked. 

The  Teamwork  Observation  Measure  (TOM)  is  an  example  of  a 
tool  that  can  be  used  in  assessing  team  skills  across  collective 
tasks.  This  instrument  was  recently  applied  in  measuring  team 
performance  during  the  planning,  contact  point,  and  attack  phases 
of  close  air  support  (CAS)  missions  (Bell,  Dwyer,  Love,  Meliza, 
Mirabella,  &  Moses,  in  preparation) .  The  same  instrument  was 
used  across  each  phase  of  CAS  for  multiple  CAS  missions, 
providing  an  opportunity  to  assess  whether  there  are  trends  in 
teamwork  skills  and  deficiencies.  Teamwork  was  assessed  in  terms 
of  the  general  areas  of  communication,  coordination,  situational 
awareness,  and  adaptability  using  the  prompts  and  rating  scales. 

Evaluating  User  Acceptance  of  Displays.  Shlechter,  Bessemer, 
Rowatt,  and  Nesselroade  (1994)  collected  comments  from  trainers 
regarding  their  opinions  of  various  UPAS  data  displays.  These 
researchers  found  that  ratings  of  displays  are  influenced  by  such 
factors  as  the  combat  mission  being  trained/evaluated,  the  time 
and  effort  required  to  use  the  display,  specific  design  features 
of  the  display  (background  contrast) ,  display  control  features 
(e.g.  fast  forward  capability) ,  rank  of  the  trainer,  echelon  to 
be  trained,  and  experience  using  the  display. 

Additional  research  is  needed  to  gain  information  about  the 
utility  of  specific  types  of  displays  as  a  function  of  key 
variables.  However,  the  work  by  Shlechter  et  al.,  (1994) 
identified  certain  capabilities  that  need  to  be  called  out  in 
requirements  documents.  First,  the  system  must  have  a  "fast 
forward"  capability  to  support  use  of  replays  during  AARs . 

Second,  the  system  must  allow  the  user  to  match  vehicle  bumper 
numbers  with  specific  icons.  Third,  users  want  access  to  line- 
of -sight  data. 

The  capability  to  fast  forward  through  replays  does  not 
necessarily  meet  the  need  of  thfe  trainer  to  quickly  identify  key 
tactical  events  and  replay j these  for  exercise  participants.  For 
example,  unless  the  system* can  move  backwards  in  exercise  history 
it  will  be  difficult  for  the  user  to  focus  in  on  critical  actions 
and  replay  these  actions  at  normal  speed.  Other  capabilities 
that  might  combine  with  fast  forwarding  to  meet  the  needs  of 
trainers  are  the  capability  to  move  directly  from  one  point  in 


21 


time  to  another  and  the  capability  to  step  forward  or  backward 
from  a  point  in  one  second  increments. 

"Out- the -window  Views" .  Many  leaders  emphasize  the  utility 
of  out -the -window  views  during  AARs.  "Out -the -window"  views 

allow  a  unit  to  see  a  replay  of  an  exercise  from  the  same 
perspective  it  had  during  an  exercise  or  from  the  enemy  s 
perspective.  As  in  the  case  of  other  types  of  AAR  displays,  we 
still  know  very  little  about  which  aspects  of  performance  can  be 
examined  most  effectively  using  this  view. 

A  weakness  in  the  control  of  some  versions  of  this  display  is 
that  it  allows  the  observer  to  get  lost  when  moving  around  the 
battlefield.  The  Simulation  Training  Integrated  Performance 
Evaluation  System  (STRIPES)  provides  the  capability  to  observe  a 
plan  view  and  "out -the -window"  view  concurrently  on  the  same 
screen.  A  plan  view  icon  shows  where  the  "out -the -window  view 
is  located  and  oriented. 

citT-atpaiRs  foT-  Nf^Tnina  Aids  Make  Teachina/Trainincr  Ppiri.ta- 
Another  area  of  display  enhancement  concerns  focusing  the 
attention  of  units  on  specfic  training  points.  In  the  case  of ^ 
the  UPAS,  this  was  accomplished  by  giving  the  user  the  capability 
to  save  a  screen  and  add  up  to  two  lines  of  comments  for  display 
with  the  screen.  For  example,  a  UPAS  Battle  Flow  Display  showing 
a  trace  of  the  movement  of  individual  vehicles  might  show  platoon 
vehicles  wandering  around  for  many  minutes  trying  to  withdraw  to 
an  alternate  position.  A  trainer  comment  like  "withdrawal  before 
selecting  alternative  positions  and_conducting  reconnaissance  of 
routes"  for  this  Battle  Flow  identifies  the  tactical  event  and 
offers  a  potential  diagnosis  of  a  performance  problem. 

Another  method  of  enhancing  AAR  displays  was  suggested  at  the 
first  AAR  Conference.  This  suggestion  involved  giving  trainers 
the  capability  to  draw  lines  and  figures  on  AAR  aids  to  highlight 
or  supplement  key  information  in  the  display.  This  capability  is 
often  referred  to  as  the  "Maddeh  Pen" . 

I 

'  Diffprences  in  Data  Available  Across  Training  EnvirQnment.a- 
The  data  available  to  prepare  AAR  aids  differ  among  the  training 
environments.  Information  about  the  status  of  entities, 
including  information  about  where  the  gun  tube  of  a  tank  is 
pointed  at  a  specific  moment,  is  readily  available  and  accessible 
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in  the  virtual  environment.  In  the  live  environment,  update 
rates  on  such  aspects  of  entity  status  as  vehicle  location  are 
slower  than  in  the  virtual  world,  and  some  types  of  information 
(speed  of  entities  and  orientation  of  gun  tubes)  are  unlikely  to 
be  provided  in  the  near  future.  Similarly,  the  constructive 
environment  does  not  play  individual  entities  (except  when 
attempting  to  link  to  the  other  environments  in  a  STOW 
application) . 

The  UPAS  was  intended  to  help  integrate  performance 
measurement  across  the  live  and  virtual  environments  by 
patterning  the  UPAS  relational  database  after  the  relational 
tables  used  for  thei-NTC  data  archives.  Using  the  exact 
relational  table  structure  was  not  considered  to  be  a  viable 
option,  because  it  would  preclude  the  use  of  data  unique  to  the 
virtual  environment  represented  by  SIMNET.  The  approach  taken 
was  to  adopt  all  of  the  table  and  column  names  from  the  NTC 
Archive  structure  and  enhance  certain  tables  with  columns 
containing  data  unique  to  the  SIMNET  environment.  Table  2  shows 
the  UPAS  Ground  Player  Location  Table  indicating  the  columns 
unique  to  SIMNET  with  an  asterisk  (*) - 

A  common  database  structure  was  adopted  under  the  assumption 
that  software  developed  to  analyze  data  in  one  environment  could 
be  rapidly  ported  for  use  in  the  other  environment .  This 
approach  at  developing  a  common  database  design  did  not  prove 
useful.  The  most  significant  problem  was  that  position  location 
data  are  reported  in  different  formats  between  SIMNET  and  NTC 
environments,  requiring  data  transformations  to  put  location  data 
into  the  correct  format.  ARI  was  able  to  examine  SIMNET  data 
from  one  exercise  using  a  utility  developed  to  analyze  NTC  data, 
but  this  required  manually  transforming  position  location  data. 
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Table  2. 


Contents  of  the  UPAS  Ground  Player  Location  Table 

o  Time  of  Vehicle  Status  Update 
o  Player  Bumper  Number 
o  Logical  Player  Number 

o  Position  of  vehicle  expressed  in  terms  of  XYZ  coordinates 

o  Position  of  Vehicle  expressed  in  terms  of  XY  coordinates 
relative  to  the  origin  of  the  terrain  database* 

o  Vehicle  speed  (in  kilometers/hour)* 

o  Vehicle  azimuth  (in  MILS)* 

o  Gun  elevation  (in  MILS)* 

o  Turret  azimuth  (in  MILS)* 

o  Engine  speed  (kilometers/hour) * 

o  Odometer  reading  (in  kilometers) * 

o  Rounds  of  ammunition  remaining* 

o  Fuel  remaining  (in  liters)* 


A  second  major  problem  becomes  apparent  if  one  considers  what 
would  be  involved  in  trying  to  enhance  the  NTC  data  to  provide 
the  information  available  in  SIMNET.  Either  a  significant 
investment  would  have  to  be  made  automating  the  production  and 
communication  of  data  (e.g.,  to  provide  information  about  engine 
speed) ,  or  the  information  need  could  be  addressed  by  behavioral 
observations  (e.g.,  an  observer  reports  significant  changes  in 
engine  speed  for  a  unit  as  a  whole) .  The  problem  with  the  latter 
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approach  is  that  it  results  in  a  situation  where  one  environment 
uses  qualitative  data  while  another  uses  quantitative  data. 

By  limiting  AAR  aids  for  each  echelon  to  those  supported  by 
data  elements  common  to  all  three  environments  we  are  throwing 
out  many  of  the  benefits  of  electronic  battlefields.  We  might 
make  tradeoffs  to  retain  many  of  the  benefits  of  standardization 
without  losing  the  benefits  of  the  unique  data  offered  by  each 
training  environment.  What  we  might  standardize  are  a  set  of 
unit  behavior  aspects  that  are  examined  in  each  environment . 


For  example,  an  important  aspect  of  the  behavior  of  tank 
platoons  is  whether!  vehicles  "shoot  and  scoot".  Using  "out  the 
window"  or  plan  view  replay  capabilities  in  the  virtual ^ 
environment,  a  trainer  can  show  cases  where  vehicles  failed  to 
move  to  alternate  positions  quickly  after  firing  and,  as  a 
result,  were  damaged  by  the  enemy's  returning  fire .. .making  a 
very  strong  teaching  point.  A  trainer  in  the  live  environment 
might  make  the  same  point  by  asking  tank  commanders  to  raise 
their  hands  if  they  were  hit  by  enemy  fire,  keep  their  hands  up 
if  they  were  engaging  the  enemy  preceding  their  receipt  of  fire, 
and  keep  their  hands  up  longer  if  they  had  failed  to  "shoot  and 
scoot".  This  approach  does  not  provide  objective  documentation 
of  "shooting  and  scooting"  but  it  does  help  to  focus  attention  on 
critical  behaviors. 

AAR  gvstam  Requirements  Derived  from  Force  X^l-  The  Force 
XXI  effort  is  concerned  with  leveraging  information  technology  to 
enhance  the  capabilities  of  warfighters,  and  an  important  part  of 
the  leveraging  process  is  the  exploitation  of  modeling  and 
simulation  to  support  training.  One  portion  of  Force  XXI 
involved  the  use  of  a  survey  to  identify  requirements  for  an  AAR 
system  as  a  function  of  such  variables  as  intended  application 
(training  feedback  versus  research)  and  echelon  (Behringer, 
Brigance,  Buckley,  Hukill,  McDonald,  &  Sayre,  1995).  The 
capabilities  identified  in  this  survey  provide  additional  input 
for  a  STAARS  ORD.  To  date,  the'  number  of  individuals  responding 
to  the  survey  is  too  small. to  address  all  of  the  information 
needs  the  survey  was  desigfied  to  meet . 
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T.^^gsn-ns  T.^^T-nPrl  trnm  the  F-vp^rimental  Force  (EXFOR) 

£j;earam.  a  distinctive  feature  c£  the  AAR 
Che  EXFOR  program  is  the  attempt  to  begin  ad  ressing 
displays  being  added  to  tactical  vehicles  as  part  of 
digitization  of  the  battlefield.  In  addition  to 
communications,  digital  transmission  of  data  and 
these  data  within  vehicles  must  be  considered  as  a 
performance  variable. 

Summary 

The  application  of  data  displays  to  support  AARs  is  a  fairly 
new  area,  and  we  have  much  to  learn  about  variables 

determining  the  utility  of  specific  types  of  displays.  Furth  , 
also  need  to  identify  new  types  of  displays  that  might  support 
the  application  of  standards  requiring  the  integration 
electronic,  planning,  communications,  terrain,  an  e  avior 
observation  data  over  tiine. 

in  addition  Co  selecting  types  of  displays  to  be  Included  in 
a  STAARS  we  are  also  faced  with  the  tasks  of  identifying 
diS^  Inrs^stem  control  features  that  will  meet  the  needs  of 
users.  We  know  that  the  capabilities  to  fast  forwar 
replays,  identify  players  based  on  bumper 
line -of -sight  between  players  are  important  con  ro 

Standardizing  AAR  aids  across  training  ^ 

echelons  helps  ensure  that  AAR  aids  will  e  rea  i  y 
to  users,  but  we  must  avoid  throwing  out  valuable 
unique  to  one  or  two  training  environments.  One  way  to  make 
aids  are  readily  interpretable  without  standardization  might 
to  educate  exercise  participants  before 
regarding  the  AAR  aids  they  are  likely  to  see  and 

of  these  aids. 


Recommendations 

The  list  below  provides  requirements  recommended 
inclusion  in  the  STAARS  ORD.  In  keeping 

philosophy,  these  requirements  describe  a  needed  capabily 
rather  than  a  specific  software  approach  to  provi  mg  e 
capability . 
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o  The  system  must  provide  the  capability  to  support  the 
implementation  of  new  types  of  data  displays  integrating 
planning  data,  terrain  data,  communications  data, 
observational  data,  and  electronic  data  streams. 

o  The  STAARS  should  allow  users  to  store  and  analyze  all 
electronic  data  produced  within  any  of  the  three  training 
environments . 

o  The  system  must  allow  the  user  to  navigate  through  replays 
by  providing  the  capabilities  to  "fast  forward" ,  move 
forward  or  backward  directly  from  one  point  in  time  to 
another,  and  hiove  forward  or  back  in  one  second  increments. 

o  The  system  must  allow  the  user  to  identify  specific 
entities  and  units. 

o  The  system  should  provide  navigational  assistance  for  "out 
the  window"  displays  to  keep  the  user  from  getting  lost. 

o  The  system  should  allow  users  to  type  in  comments,  draw 
lines,  and  draw  figures  on  AAR  aids. 

o  The  system  should  support  the  storage,  retrieval,  and 

display  of  a  library  of  AAR  aids  using  text,  graphics,  and 
figures  to  describe  alternative  tactics,  technigues,  and 
procedures  to  support  the  "a  way"  concept . 

There  are  a  number  of  research  actions  that  need  to  be 
addressed  to  refine  STAARS  requirements  or  support  implementation 
of  the  STAARS  concept.  The  actions  are  to: 

o  examine  costs  and  benefits  of  alternative  operational 
definitions  of  the  concept  of  "standardizing  AAR  aids 
within  echelons" ; 

o  define  and  defend  a  standh.rd  set  of  AAR  aids  for  a  tank 
platoon  or  mechanized  infantry  platoon; 

o  identify  standards  or  types  of  standards  for  which  improved 
data  displays  are  needed; 
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o  assess  factors  determining  acceptability  of  display  types 
to  various  user  groups;  and 

o  examine  alternative  methods  for  making  sure  that  AAR  aids 
are  readily  interpretable. 


Tools  to  Help  Trainers  Prepare  for  AARs 


Background 

An  AAR  system  should  help  ensure  that  displays  are  ready  to 
support  AARs  as  soon  as  possible  after  the  end  of  an  exercise 
(ENDEX) .  This  is  especially  true  for  AARs  conducted  for  lower 
echelons,  because  the  results  of  lower  echelon  AARs  provide  input 
for  the  later  conduct  of  higher  echelon  AARs .  In  the  Fort  Knox 
Mounted  Warfare  Training  Simulation  Center  (MWTSC) ,  for  example, 
the  goal  is  to  begin  conducting  platoon- level  AARs  ten  minutes 
after  ENDEX. 

The  job  of  preparing  to  conduct  AARs  includes  deciding  what 
teaching/ training  points  need  to  be  made,  deciding  what  AAR  aids 
can  be  used  to  help  illustrate  points,  developing  questions  to  be 
used  in  guiding  unit  discussions  during  the  AAR,  and  creating  AAR 
aids.  This  work  must  also  be  coordinated  with  the  exercise 
management  and  control  functions  of  trainers  to  ensure,  for 
example,  that  AAR  preparation  does  not  distract  a  trainer  from 
exercise  control  activities. 

BTAARS  and  PIS  Standards  Guidance 

The  STAARS  MNS  does  not  specify  the  functions  a  STAARS  should 
perform  to  help  a  trainer  prepare  for  AARS,  but  it  does  provide 
information  that  must  be  considered  when  developing  requirements 
for  a  STAARS  system  and  when  selecting  a  materiel  solution  to  the 
STAARS  requirements.  The  STAARS  must  avoid  duplication  in  the 
development  of  software  to  support  the  preparation  of  data 
displays .  The  STAARS  must  work  in  a  manner  that  will  not  disrupt 
exercises.  Finally,  the  system  must  be  applicable  across 
training  environments .  The  DIS  Standards  guidance  does  not 
address  what  a  system  should  do  to  help  users  prepare  for  AARs . 

History  and  Status  of  ResearC-h 

Historical  Persnective One  of  the  key  concerns  driving 
development  of  the  UPAS  was  the  shortage  of  tools  to  support  AARs 
in  the  SIMNET  training  environment  (Goldberg  &  Meliza,  1993) . 
These  tools  included  one  Plan  View  display  and  one  Stealth  (out 
the-window)  view  in  situations  where  as  many  a  five  exercises 
might  be  conducted  concurrently.  These  sites  did  not  support  the 
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capability  to  provide  statistical  summaries  such  as  the  number  of 
rounds  fired  by  each  crew  in  a  platoon. 

By  adopting  a  personal  computer  (PC)  as  the  hardware 
platform,  we  gained  the  capability  to  distribute  AAR  systems  down 
to  platoon  level  at  a  low  cost.  However,  in  comparison  with  work 
stations,  we  lost  speed  and  multi-tasking  capabilities.  One  of 
the  major  problems  encountered  in  applying  the  UPAS  as  an  AAR 
tool  was  that  it  took  too  much  time  to  prepare  AAR  aids 
(Shlechter  et  al.,  1994;  Meliza,  Bessemer,  &  Tan,  1994).  For 
platoon  exercises,  trainers  want  to  start  AARs  within  about  ten 
minutes  after  ENDEX.  Due  to  the  fact  that  UPAS  is  a  PC-based 
system,  it  could  only  perform  one  function  at  a  time.  During 
exercises,  UPAS  was'  fully  occupied  with  network  data  collection 
and  could  not  be  used  to  support  exercise  monitoring  or  AAR  aid 
preparation.  After  data  collection,  a  substantial  amount  of  time 
was  required  to  generate  a  second  by  second  index  file  of  network 
data  and  load  network  data  into  a  relational  database  to  support 
preparation  of  graphs  and  tables.  The  time  required  to  generate 
index  files  and  load  data  tables  often  exceeded  ten  minutes . 

Another  problem  with  using  UPAS  as  an  AAR  system  was  that  it 
left  the  job  of  selecting  and  preparing  AAR  aids  up  to  trainers. 
The  work  left  up  to  trainers  included:  assessing  the  major  unit 
strengths  and  weaknesses;  selecting  the  types  of  displays  to  use 
in  illustrating  outcomes  and  guiding  a  unit  towards  diagnosing 
their  own  strengths  and  weaknesses;  selecting  appropriate  display 
parameters  (e.g.,  time,  area,  level  of  magnification);  and 
developing  questions  or  comments  to__be  used  with  each  display. 

The  move  to  workstations  reduced  the  time  required  to  perform 
certain  AAR  preparation  tasks  and  provided  the  opportunity  to  add 
tools  for  helping  trainers  to  prepare  AAR  aids .  UPAS  data 
displays  have  been  ported  to  work  stations  in  two  separate 
efforts  to  address  the  problems  we  discovered  in  UPAS  usage.  At 
roughly  the  same  time,  other  workstation-based  AAR  systems  were 
also  being  developed  to  support*  AARs  in  the  virtual  and 
constructive  environments., 

STRICOM  funded  an  effort  by  Loral  Advanced  Distributed 
Simulation  to  port  UPAS  data  displays  and  functions  to  a  Silicon 
Graphics  workstation.  This  effort  was  performed  by  integrating 
UPAS  displays  with  existing  DIS  Tools  software  that  included  two 
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dimensional  and  “out  the  window"  views  of  the  battlefield. 

STRIPES  can  be  used  to  monitor  an  exercise  from  a  plan  view 
and/or  "out  the  window"  view  at  the  same  time  that  exercise  data 
are  being  collected.  Due  to  the  fact  that  STRIPES  runs  on  a 
workstation,  the  time  to  load  the  relational  database  is  greatly 
reduced.  However,  as  in  the  case  of  UPAS,  the  job  of  preparing 
an  AAR  begins  at  ENDEX. 

Antomatftd  and  Semi -automated  Generation  of  AAR 

Lockheed  Martin  demonstrated  a  system  at  the  second  AAR 
conference  that  automated  production  of  AAR  aids.  This  system 
employed  a  workstation  to  collect  exercise  data.  At  ENDEX,  a 
software  routine  is  used  to  load  AAR  aids  to  a  floppy  file  for 
delivery  on  a  PC.  The  beginning  and  end  points  of  the  time 
covered  by  each  aid  was  decided  by  automated  analysis  of  the 
network  data  stream  to  identify  when  certain  tactical  events 
occurred  (e.g.,  the  time  of  the  first  friendly  direct  fire).  The 
data  displays  included  tables,  graphs,  and  short  animated  plan 
views.  Further,  the  system  allowed  the  trainer  to  include  other 
types  of  AAR  aids,  such  as  TTPs  from  “how  to  fight"  manuals. 

LBScM  Associates  developed  the  Automated  Training  Analysis  and 
Feedback  System  (ATAFS)  to  help  trainers  prepare  and  conduct  AARs 
by  addressing  problems  identified  in  using  UPAS  (Brown,  et  alw 
1995) .  ATAFS  uses  a  knowledge  database  to  guide  the  preparation 
of  AARs  aids  automatically.  In  addition,  ATAFS  allows  users  to 
move  back  in  exercise  history  and  create  AAR  aids  manually  as  the 
system  continues  to  collect  exercise  data.  The  result  of  these 
two  enhancements  is  to  provide  an  AAR  Bin  with  automatically  and 
manually  generated  AAR  aids  at  ENDEX. 

The  automated  production  of  AAR  aids  in  ATAFS  is  guided  by  a 
knowledge  database.  To  create  a  particular  aid,  a  system  needs 
to  know  the  time  to  be  covered  by  the  aid  (e.g.,  from  14:33:00 
until  14:35:15) .  These  points  in  time  are  dependent  upon  when 
tactical  events  occur.  The  devfelopers  of  ATAFS  realized  that  the 
occurrence  of  some  of  these  events  could  be  measured 
automatically  by  an  AAR  system  through  analysis  of  the  network 
data  stream.  The  time  of  other  events  is  better  identified  by 
having  a  human  look  for  and  report  the  occurrence  of  these  events 
(e.g.,  a  platoon  leader  issues  an  order) .  ATAFS  creates  aids 
using  a  mix  of  data  stream  analysis  combined  with  operator 
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response  to  prompts  made  up  of  a  list  of  tactical  events  which 
the  trainer  is  expected  to  observe.  Figure  5  illustrates  the 
screen  a  trainer  uses  when  monitoring  an  exercise. 

Another  way  in  which  the  ATAFS  helps  trainers  prepare  timely 
AARs  is  to  allow  users  to  move  back  in  exercise  history  and 
create  aids  manually  as  the  ATAFS  continues  to  collect  data.  The 
VCR- like  controls  at  the  upper  left  corner  of  the  ATAFS  screen 
can  be  used  to  move  back  in  history,  or  return  to  real  time,  as 
the  ATAFS  continues  to  collect  data.  If  the  user  wants  to  create 
an  aid,  he  or  she  selects  one  of  the  buttons  at  the  upper  right 
hand  portion  of  the  screen,  such  as  "Plan  View" .  The  user  then 
selects  the  button  pnce  to  indicate  the  time  when  a  Plan  View 
should  begin  and  a  second  time  to  indicate  when  it  should  end. 

At  ENDEX,  the  trainer  can  go  to  the  ATAFS  AAR  Aid  Bin  and  review 
the  aids  that  have  been  automatically  and  manually  created.  Each 
of  the  automatically  generated  aids  includes  a  list  of  discussion 


32 


questions  that  might  be  used  by  the  trainer  to  help  insure  the 
AAR  discussion  process  will  lead  towards  an  explanation  for  the 
unit's  performance  (see  Figure  6). 
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Figure  5 .  Sample  exercise  monitor  screen  for  the  ATAFS . 
(Copyrighted  material  reprgduced  with  permission  of  LB&M 
Associates.) 
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Fire  Fight  AAR  Aid 


Figure  6.  Sample  ATAFS  AAR  aid  showing  discussion  points. 
(Copyrighted  material  reproduced  with  permission  of  LB&M 
Associates . ) 

Frfitina  AAR  Aids.  The  ATAFS  system  allows  trainers  to 
perform  two  types  of  editing  functions.  First,  trainers  can  add 
or  modify  discussion  questions  or  comments  at  the  right  hand  side 
of  each  aid,  and  they  can  name  or  rename  the  tile  of  the  aid  or 
the  tactical  events  defining  thfe  aid.  Second,  trainers  can 
modify  the  period  of  time  covered  by  an  aid.  This  latter 
capability  is  useful  if  the  original  time  period  leads  to  a 
situation  where  part  of  a  radio  communication  is  cut  off  and  the 
trainer  wants  to  expand  the  time  covered  to  include  the  entire 
communication.  The  latter  capability  is  also  useful  when  an 
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unexpected  critical  tactical  event  occurs  that  a  trainer  wants  to 
include  in  the  aid. 

It  is  important  to  point  out  that  the  ATAFS  creates 
definitions  of  AAR  aids  rather  than  AAR  aids  per  se.  That  is, 
aids  are  defined  by  time  and  any  changes  the  user  makes  to  the 
standardized  list  of  discussion  points  or  labels.  From  the 
perspective  of  the  user  it  appears  that  the  entire  aid  is  saved, 
because  the  aid  appears  on  screen  so  fast  after  it  is  selected. 

.qavina  AAR  Aids .  Earlier  versions  of  the  ATAFS  failed  to 
save  the  definitions  of  AAR  aids  that  had  been  automatically  or 
manually  created  when  the  user  exited  the  exercise  database.  The 
capability  to  save  the  definitions  of  AAR  aids  within  the 
exercise  database  was  not  considered  to  be  necessary ,  because 
these  aids  can  be  saved  by  downloading  them  to  a  VCR  tape.  The 
most  serious  consequence  of  failing  to  save  AAR  aids  in  the 
exercise  database  is  that  all  of  the  aids  are  lost  if  the  system 
"crashes"  before  the  AAR  is  conducted. 

Failure  to  save  the  definitions  of  AAR  aids  also  reduced  the 
utility  of  ATAFS  as  an  AAR  research  tool.  Saving  of  the 
definitions  allows  researchers  to  examine  exercise  databases  to 
answer  such  questions  as:  to  what  extent  do  users  supplement 
automatically  generated  aids  with  manually  created  aids?  What 
portion  of  the  aids  are  actually  used  for  AAR? 

Oivina  the  User  the  Ability  to  Automate  AAR  Aid  Pygduct3,dn • 
The  ATAFS  project  includes  the  development  of  an  authoring  tool 
(ATAFS-AT)  to  allow  non-programmers  to  modify  the  sets  of  aids 
created  automatically  for  a  mission  and/or  automate  AAR  aid 
production  for  additional  missions  (Brown,  et  al . ,  1995) . 

Figures  7  and  8  help  to  illustrate  how  this  tool  will  work.  To 
automate  the  production  of  an  aid,  the  user  must  identify  the 
type  of  aid,  the  type  of  tactical  events  triggering  the  beginning 
and  ending  of  the  time  covered  by  an  aid  (unit  crossing  of  a 
line,  unit  entering  an  area,  a  firing  event,  a  battle  damage 
event,  or  an  event  that  mu^t  be  identified  by  a  trainer  through 
response  to  a  screen  prompt) ,  the  name  of  the  aid,  a  description 
of  the  triggering  events,  and  discussion  points. 

After  selecting  icons  representing  trigger  types,  users 
select  the  specific  parameters.  For  example,  if  the  icon  for 
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crossing  a  line  is  selected,  the  user  must  select  first  vehicle, 
last  vehicle,  or  center  of  mass  of  platoon  to  define  the  criteria 
ATAFS  will  use  in  deciding  when  the  line  has  been  crossed.  The 
user  must  also  select  among  a  group  of  options  to  identify  the 
type  of  line  to  be  crossed,  such  as  the  Line  of  Departure,  a 
phase  line,  or  a  boundary.  If  the  trigger  is  a  screen  prompt  the 
user  may  select  an  existing  prompt  or  create  a  new  one. 
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Figure  7.  Creating  or  modifying  an  automated  AAR  aid  through 
icon  selection.  (Copyrighted  material  reproduced  with  permission 
of  LB&M  Associates . ) 
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Figure  8 .  Creating  or  modifying  an  automated  AAR  aid  by 
defining  icon  parameters.  (Copyrighted  material  reproduced 
with  permission  of  LB&M  Associates.) 

Linking  AAR  Aids  to  Exercise  Outcomes.  Automated  support  of 
AAR  aid  preparation  should  go  bfeyond  providing  a  standardized  set 
of  data  displays  for  specific  collective  tasks.  Participants  in 
the  first  AAR  Conference  nCted  the  need  for  tailoring  AAR  aids  to 
fit  the  outcome  of  exercises  (Goldberg,  1994) .  While  the 
occurrence  or  non- occurrence  of  certain  tactical  events 
influences  the  set  of  candidate  AAR  aids  produced  for  an  exercise 
by  the  ATAFS  knowledge  database,  this  does  not  guarantee  that 
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every  aid  will  be  useful.  ATAFS  might  produce  eighteen  AAR  aids, 
but  usually  only  three  or  four  of  these  are  needed  to  help  make 
the  points  the  trainer  wants  to  make.  Additional  tools  are 
required  to  help  trainers  select  among  candidate  AAR  aids. 

Focusing  on  Specific  Behaviors.  There  are  a  large  number  of 
performance  standards  that  might  be  applied  to  a  unit  during 
exercises.  In  many  cases,  limitations  in  the  time  available  to 
conduct  an  AAR  preclude  providing  a  unit  with  feedback  on  how  it 
performed  with  respect  to  every  standard.  Another  function  of  an 
AAR  system  should  be  to  help  a  trainer  focus  on  selected  aspects 
of  unit  performance . 

For  battalion  task  force  level  AARs  at  the  NTC,  a  strategy  is 
employed  to  help  focus  the  AAR  on  specific  aspects  of  unit 
performance.  That  strategy  involves  focusing  on  aspects  of  unit 
behavior  that  contributed  directly  to  the  outcome  of  a  mission 
(Meliza,  Sulzen,  Atwood,  &  Zimmerman,  1987) .  This  approach  is 
expected  to  help  motivate  units  to  take  corrective  actions  by 
demonstrating  the  importance  of  the  deficiency  to  unit  success. 
This  link  provides  a  stronger  motivation  than  pointing  out  that 
the  performance  of  the  unit  was  not  in  accordance  with  Army 
doctrine . 

Another  approach  to  focusing  on  selected  aspects  of 
performance  is  to  consider  errors  which  tend  to  show  up  across  a 
large  number  of  units.  For  example,  if  it  were  found  to  be  the 
case  that  armor  and  infantry  platoons  frequently  move  in  the 
immediate  vicinity  of  friendly  vehicles  or  units  that  have  just 
been  successfully  engaged  by  the  enemy,  this  would  be  an  aspect 
of  behavior  that  a  trainer  should  be  prepared  to  check. 

Tnt<=>aratina  the  AAR  Preparation  System  with  Sy:eycigg  Plahnincf 
anH  contr-nl  .qvstams .  The  job  of  preparing  to  conduct  an  A^  is 
tied  to  the  exercise  planning  and  control  systems .  When  highly 
structured  scenarios  are  employed,  a  trainer  can  better 
anticipate  when  key  exercise  evfents  will  occur  and  use  AAR  aids 
to  focus  on  these  events.  jWhen  highly  unstructured  scenarios  are 
employed,  trainers  are  at  a  distinct  disadvantage  in  terms  of  AAR 
preparation. 

In  most  cases  a  trainer  serves  both  exercise  control  and 
AAR  preparation  functions  during  exercises.  Exercise  control 
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functions  include  acting  as  the  higher  headquarters  to  the  unit, 
monitoring  actions  to  make  sure  the  tactical  situation  supports 
the  training  objective,  and  intervening  in  the  tactical  situation 
to  help  insure  the  exercise  provides  an  effective  training 
opportunity.  To  the  extent  that  the  AAR  system  and  exercise 
control  system  compete  for  the  attention  of  the  trainer,  AARs  and 
exercise  control  suffer. 

Two  examples  of  the  lack  of  integration  between  AAR  and 
exercise  control  systems  can  be  provided  based  upon  experiences 
within  the  Fort  Knox,  KY,  Reserve  Component  Virtual  Training 
Program  (RCVTP) .  First,  the  UPAS  was  added  to  the  RCVTP  in  a 
situation  where  the? exercise  control  system  was  composed  of  a 
modular  semi -automated  force  (ModSAF)  work  station  for 
controlling  the  behavior  of  enemy  and  friendly  computer  generated 
forces  and  emulating  supporting  fires.  In  this  situation,  a 
ModSAF  Plan  View  and  an  "out  the  window"  view  were  available  to 
help  monitor  unit  performance.  Two  individuals  were  involved  in 
the  conduct  of  training;  one  to  control  the  ModSAF  during 
exercises  and  the  AAR  displays  during  the  exercise,  and  a  second 
to  serve  as  the  trainer.  An  immediate  problem  was  that  the  time- 
consuming  job  of  loading  control  measure  graphics  data  had  to^ 
performed  separately  for  ModSAF  and  UPAS.  Having  the  capability 
to  load  graphics  into  one  system  and  then  copy  the  resulting  data 
into  the  other  would  improve  training  efficiency.  It  should  not 
be  too  surprising  in  this  case  that  the  RCVTP  trainers  used  the 
same  tools  they  employed  in  monitoring  exercises  (the  ModSAF  plan 
view  and  "out  the  window"  view)  to  provide  AAR  aids  rather  than 
trying  to  support  a  second  system.  _ 


The  second  example  of  an  integration  problem  involves 
competition  during  exercises  between  the  ATAFS  AAR  system  and  the 
exercise  control  system  described  above.  The  UPAS  did  not 
compete  with  exercise  control  during  exercises  for  the  simple 
reason  that  all  the  UPAS  can  do  at  that  time  is  collect  data. 
ATAFS  helps  trainers  to  begin  preparing  AARs  during  exercises, 
increasing  the  potential  for  cohflicts  between  the  exercise 
control  and  AAR  systems.  To  employ  the  semi-automatically 
generated  ATAFS  AAR  aids,  trainers  must  respond  to  screen 
prompts,  and  use  of  the  prompts  requires  monitoring  the  radio 
net .  The  result  is  to  add  a  third  monitor  that  can  be  used  to 
observe  the  exercise  and  a  second  set  of  speakers  providing  radio 
communications.  In  an  ideal  world,  the  ModSAF  operator  would  use 
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the  ModSAF  workstation,  and  the  trainer  would  use  the  ATAFS  to 
monitor  the  exercise;  however,  the  radio  communications  over  the 
ATAFS  (and  all  movement  and  firing  events  displays  on  the  ATAFS 
screen)  are  roughly  two  to  three  seconds  behind  real  time  due  to 
the  fact  that  ATAFS  processes  data  before  it  is  displayed.  The 
delay  has  no  effect  on  AAR  aid  preparation,  but  it  makes  it 
difficult  to  use  ATAFS  radio  communications  as  part  of  the 
exercise  control  loop.  A  second  problem  is  that  trainers  want  to 
use  the  "out  the  window"  view  for  both  exercise  control  and  AAR 
functions,  causing  the  trainer  to  have  to  interact  with  two 
workstations  during  AARs. 

STRICOM  is  awarfe  of  the  need  to  integrate  the  various 
workstations  used  in  the  DIS  environment  (Watson  &  Schow,  1995, 
Butler  &  Wiehagen,  1995) .  Further,  STRICOM  is  preparing  to 
initiate  an  effort  to  develop  a  testbed  for  integrating  system 
components . 

Training  TT.gp-rs  of  a  STAARS .  Given  the  reductions  in  military 
funding  it  is  unlikely  that  every  site  where  AAR  systems  are ^ 
employed  will  have  personnel  permanently  assigned  to  0/C  duties. 
Instead,  the  trainer  conducting  the  AAR  will  often  be  from  the 
unit  being  trained,  and  the  unit  might  have  limited  access  to  A^ 
system  hardware  and  software.  For  example,  a  National  Guard  Unit 
might  have  access  to  a  mobile  SIMNET  system  once  or  twice  a  year 
over  weekends.  Fielding  an  AAR  system  that  can  be  used  without 
substantial  hands-on  practice  is  a  major  challenge. 

The  Defense  Advanced  Research  Projects  Agency  (DARPA) 
Simulation  in  Training  for  Advanced  Readiness  (SIMITAR)  Program 
has  taken  a  major  step  in  addressing  this  challenge  by  funding 
the  development  of  a  computer-based  training  (CBT)  program  for 
users  of  the  ATAFS  combined  with  trial  fielding  the  CBT  and  ATAFS 
system  at  a  variety  of  Army  National  Guard  (ARNG)  training  sites. 
These  sites  include  mobile  SIMNETs  in  the  Northwest  and  Southeast 
as  well  as  fixed  sites.  The  fixed  sites  include  ARNG  armories 
and  the  MWTSC  at  Fort  Knox,  KY.'  The  extent  to  which  individuals 
conducting  AARs  will  have  the  opportunity  to  gain  experience 
using  an  AAR  system  increases  going  from  mobile  SIMNETs  to 
armories  to  the  MWSTSC.  System  fielding  at  these  diverse  sites 
provides  a  testbed  for  addressing  many  research  issues  concerning 
preparation  of  trainers  to  employ  AAR  systems. 
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Ths  foirina.1  prog^ratn  of  instmction  fo^r  ATAFS  users  includes 
approximately  two  hours  of  CBT  delivered  on  a  multi -media 
personal  computer  combined  with  two  hours  of  hands-on  practice  in 
which  members  of  a  two-person  team  alternate  taking  the  roles  of 
operator  and  informal  assistant.  Training  is  conducted  in  this 
manner  because  the  members  of  the  team  help  to  coach  each  other. 
The  hands-on  session  involve  the  replay  of  network  and  radio 
communications  data  from  a  previous  exercise  in  a  manner  that 
simulates  realtime  data  collection. 

ATAFS  installation  and  user  training  have  been  conducted  at 
four  sites,  and  participants  were  confident  of  their  ability  to 
use  the  system  after  an  hour  or  less  of  hands-on  training.  There 
have  been  problems  with  user  interactions  due  in  large  part  to 
the  fact  that  few  trainers  are  familiar  with  critical  differences 
between  PCs  and  workstations.  Additional  software  is  required  to 
help  trainers  perform  certain  routine  functions  on  the  work 
station,  such  as  deleting  exercise  directories. 

Summary 

Moving  from  a  PC  to  a  workstation  environment  enables  speed 
and  multi-tasking  capabilities  to  reduce  the  time  required  to 
prepare  AAR  aids.  Additional  reductions  in  time,  and  support  of 
the  trainer's  decision  making  process,  have  been  gained  by 
automating  the  preparation  of  AAR  aids.  The  completely  automated 
approach  relies  on  software  analysis  of  the  network  data  stream, 
while  the  semi -automated  approach  uses  trainer  reactions  to  on 
screen  prompts  to  identify  when  tactical  events  occur  that  are 
not  part  of  the  network  data  stream  (e.g.,  when  a  platoon  reports 
contact  to  the  company  commander) . 

Additional  work  is  required  if  the  AAR  system  is  to  provide 
trainers  with  further  assistance  by  helping  consider  how  exercise 
outcomes  influence  the  utility  of  candidate  AAR  aids.  We  have 
gone  beyond  the  point  of  creating  a  fixed  set  of  aids  for  a 
particular  collective  task  to  the  point  of  tailoring  the  output 
of  aids  to  fit  the  tactical  events  that  actually  occur  in  a 
specific  instance  of  task  performance.  We  need  to  go  a  little 
further  and  help  the  trainer  select  among  these  candidate  aids. 
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As  we  attempt  to  help  trainers  prepare  AAR  aids,  we  must ^ 
consider  the  need  to  integrate  the  AAR  system  with  the  exercise 
planning  and  control  systems .  We  want  an  AAR  system  to  help  a 
trainer  monitor  and  control  the  execution  of  an  exercise  rather 
than  distracting  trainers  from  these  duties.  STRICOM  is 
initiating  a  testbed  concerned  with  integrating  various  training 
system  components  in  the  DIS  environment,  and  integration  of 
exercise  control  and  AAR  functions  should  be  a  key  part  of  this 
effort . 

Training  users  to  employ  an  AAR  system  is  a  concern,  because 
time  and  other  resources  required  to  conduct  this  training  are 
severely  constrained.  LB&M  Associates  has  developed  a  CBT 
program  for  the  ATAFS  that  appears  to  prepare  most  trainers  to 
employ  this  system  to  create  automated  and  manual  aids  after  two 
hours  of  CBT  followed  by  an  hour  or  so  of  hands-on  practice  with 
the  actual  system.  The  availability  of  this  training  system  in  a 
variety  of  training  environments  enables  research  on  training 
personnel  to  employ  AAR  systems . 

Recommendations 

Items  that  might  be  included  in  a  STAARS  ORD  are  listed 
below. 

o  The  system  should  increase  rather  than  reduce  a  trainer's 
capability  to  perform  exercise  control  functions  during 
exercises . 

o  The  system  should  automate  the  production  of  candidate  AAR 
aids  and  allow  the  trainer  to  select  those  he/she  considers 
to  be  relevant  to  a  specific  exercise. 

o  The  system  should  give  the  trainer  the  capability  to 
supplement  automatically  produced  aids  with  manually 
generated  aids . 

o  The  system  should  help  the  user  select  among  candidate  AAR 
aids,  in  cases  where 'users  want  such  help. 

o  The  system  should  allow  the  user  to  store  AAR  aids  or 
definitions  of  AAR  aids. 
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o  The  system  should  allow  the  user  to  modify  the  time  period 
covered  by  each  aid. 

There  are  a  number  of  research  and  development  issues  that 
need  to  be  addressed  to  support  the  design  and  implementation  of 
AAR  systems  that  can  provide  high  quality  AAR  aids  without 
requiring  excessive  inputs  from  trainers.  Research  and 
development  needs  are  listed  below. 

o  Develop  methods  for  selecting  AAR  aids  that  fit  specific 
exercise  outcomes . 

o  Develop  strategies  for  focusing  on  specific  behaviors 

during  exercises  and  test  the  comparative  utility  and  user 
acceptance  of  the  strategies. 

o  Use  ATAFS  as  a  test  case  for  assessing  the  amount  of  CBT 
and  hands-on  training  required  to  prepare  leaders  to  employ 
an  AAR  system. 
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Creating  a  User  Friendly  AAR  System 


Background 

Systems  that  are  inflexible  or  lack  internal  consistency  in 
terms  of  procedures  frustrate  users.  If  the  system  will  not 
support  the  activity  a  user  wants  to  perform,  then  the  user  will 
often  have  to  complain  to  the  software  developer  and  wait  for  the 
next  version  of  the  software  to  employ  the  desired  capability. 

If  the  system  lacks  internal  consistency  in  terms  of  the  way 
information  is  presented  or  the  way  a  user  is  expected  to  provide 
input,  then  it  will  be  difficult  to  complete  initial  and 
sustainment  training  on  the  system,  and  the  error  rate  will  often 
remain  too  high. 

As  mentioned  earlier  in  this  report,  we  have  limited 
experience  using  AAR  aids  to  support  AARs  and  we  can  expect  the 
process  of  learning  lessons  about  AAR  tools  to  continue  over  the 
next  few  years.  We  want  to  avoid  specifying  the  exact  design 
features  of  STAARS  data  collection,  analysis,  and  AAR  preparation 
tools  too  early  in  a  manner  that  will  prevent  users  from 
modifying  tools  in  response  to  lessons  learned. 

STAARS  and  PIS  Standards  Guidance 

According  to  the  STAARS  MNS,  STAARS  will  provide  standardized 
user  definable  products  incorporating  playback  capability, 
C4I/video  products,  access  to  doctrinal  resources,  statistical 
products,  terrain  analysis,  and  trainer  observations. 

The  expression  "user  definable"  indicates  that  providing  users 
with  the  capability  to  create  and  modify  AAR  aids  is  a  high 
priority.  Further,  STAARS  must  be  flexible  enough  to  fit  the 
constructive,  live,  and  virtual  environments.  Creating  software 
that  allows  users  to  create  and  modify  AAR  aids  without  going 
back  to  the  software  developer  to  reprogram  the  system  is  also 
supportive  of  the  general  Warfighter  XXI  goal  of  reducing 
software  development  costs  by  aVoiding  duplication. 

I 

The  draft  DIS  Standard ’ concerning  exercise  management  and 
feedback  promotes  flexibility  by  identifying  capabilities  that 
have  user  selectable  features  (Institute  for  Simulation  and 
Training,  1995) .  Examples  include  selection  of  PDUs  to  be 
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collected,  selection  of  features  to  be  displayed  in  plan  view 
displays,  and  the  capability  to  customize  graphs,  tables,  and 
timelines • 

History  and  Status  of  Research 

Hystem  Flexibility.  One  of  the  most  important  lessons 
learned  during  refinement  of  UPAS  software  is  that  the  user  must 
be  provided  with  the  capability  to  control  the  contents  of  data 
displays.  Most  of  the  revisions  made  in  UPAS  software  were  made 
in  order  to  provide  the  user  with  greater  flexibility  to  control 
AAR  displays  (Meliza,  Bessemer,  and  Tan,  1994) .  This  same  lesson 
was  relearned  in  the  development  of  the  STRIPES  and  ATAFS 
workstations.  To  the  extent  that  users  can  modify  data  displays 
without  the  assistance  of  a  programmer,  the  system  is  considered 
flexible . 

The  early  prototype  of  the  UPAS  was  developed  with  the 
knowledge  that  trainers  and  researchers  could  not  specify  the 
data  summary  graphs  and  tables  they  would  need.  Therefore,  the 
UPAS  includes  editors  that  allow  non -programmers  to  add  or  modify 
graphs  and  tables.  This  makes  it  possible  to  revise  the  displays 
in  response  to  user  feedback  or  in  preparation  for  applying  the 
system  to  a  new  training  or  research  objective.  Neither  the 
STRIPES  nor  the  ATAFS,  in  their  initial  versions,  allowed  users 
to  add  or  modify  graphs  and  tables  without  the  aid  of 
programmers , 

At  the  third  AAR  Conference,  representatives  of  the  United 
Kingdom  Center  for  Defence  Analysis  described  data  summary  graph 
and  table  requirements  to  support  SIMNET  training  in  the  near 
term  and  training  with  the  United  Kingdom  CATT  in  the  longer 
term.  The  variety  of  displays  goes  beyond  what  is  currently 
available  in  STRIPES,  ATAFS,  and  the  CCTT  AAR  system.  A  near 
term  solution  to  this  problem  is  being  tested  by  modifying 
STRIPES  to  download  data  to  a  PC  during  exercises.  Once  these 
data  are  on  a  PC,  then  a  wide  vhriety  of  commercial  off-the-shelf 
(COTS)  software  can  be  used  to  create  essentially  any  type  of 
data  table  or  graph  the  user  may  want . 

A  flexibility  problem  has  also  been  encountered  in  the  design 
of  an  ATAFS-AT  to  allow  users  to  create  or  modify  the  AAR  aids  to 
be  generated  automatically  during  exercises.  In  cases  where  the 
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triggsring  event  is  one  which  reguires  analysis  of  network  data, 
the  user  is  limited  to  using  trigger  types  and  parameters  for 
which  a  programmer  has  already  written  data  analysis  routines.  A 
potential  solution  to  this  problem  would  be  to  provide  a  tool 
that  allows  non-programmers  to  collect  and  manipulate  the  data 
stream  during  exercises.  McDonough  and  Herman  (1995)  recently 
suggested  implementing  such  a  capability  for  STRIPES. 

The  flexibility  problem  often  shows  up  in  unexpected  ways. 

To  help  illustrate  the  breadth  of  the  flexibility  problem,  a  list 
of  examples  is  provided  below. 

o  During  one  of’r  the  early  applications  of  the  initial  UPAS 
Firefight  Display,  data  were  collected  from  an  exercise  in 
which  a  heavy  volume  of  fire  was  provided  by  a  weapon 
firing  25mm  rounds.  Shotlines  for  the  25mm  firing  events 
tended  to  dominate  the  displays  and  obscured  tank  firing 
events.  To  fix  this  problem,  the  software  was  revised  to 
allow  users  to  employ  sampling  plans  to  reduce  the 
shotlines  for  the  25  mm  (e.g.,  showing  shotlines  for  every 
third  round) . 

o  The  capability  of  the  ATAFS  to  generate  AAR  aids 

automatically  was  tested  in  a  situation  where  the  platoon 
was  a  six  vehicle  scout  platoon  rather  than  a  four  vehicle 
armor  or  mechanized  infantry  platoon.  The  algorithm  used 
by  the  ATAFS  to  assess  various  states  (i.e.,  when  control 
measures  are  crossed)  assumes  a  four  vehicle  platoon  and 
will  not  work  with  other  values.  There  was  a  historical 
precedent  for  this  flaw  in  that  the  UPAS  was  designed  to 
allow  the  user  to  load  vehicle  IDs  in  a  manner  that 
informed  the  system  which  of  a  maximum  of  four  vehicles 
belonged  to  each  platoon.  This  limitation  became  a  problem 
the  first  time  UPAS  was  applied  to  scout  platoons. 

o  To  expedite  the  preparation  of  data  tables  and  graphs,  the 
UPAS  extracts  data  from  ntetwork  data  packets  for  loading  a 
relational  database  management  system.  The  UPAS  data 
tables  were  patterned  after  the  NTC  archives  database  to 
support  common  analyses  of  SIMNET  and  NTC  data  and  thus  ^ 
certain  types  of  data  unique  to  SIMNET  were  not  loaded  into 
the  database.  For  example,  there  is  a  fire  packet  and 
impact  packet  generated  for  each  direct  round  fired  in 
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SIMNET,  and  each  of  these  packets  contains  a  firing  event 
number.  Firing  event  numbers  are  generated  sequentially  for 
each  entity,  and  the  same  event  number  is  used  for  the  fire 
and  corresponding  impact  packet .  Although  most  of  the 
information  in  fire  and  impact  packets  were  loaded  into  the 
UPAS  relational  database,  data  on  the  event  number  was 
viewed  as  having  no  value  and  not  included.  Subsequently 
we  found  that  these  data  were  often  critical  in  sorting  out 
the  effects  of  weapon  systems  during  exercises,  and  the 
UPAS  software  had  to  be  revised  to  load  these  data  into  the 
relational  database.  Until  this  change  was  accomplished, 
it  was  necessary  for  researchers  to  go  through  the 
extremely  tedious  process  of  examining  thousands  of  data 
packets  using 'a  UPAS  packet  reader  to  obtain  information 
about  firing  event  numbers. 

o  Not  all  lessons  learned  involve  bad  news.  An  unexpected 
benefit  of  using  a  relational  database  combined  with  an 
editing  system  to  support  data  graph  and  table  preparation 
is  that  it  allows  one  to  integrate  new  types  of  information 
into  data  displays  without  formal  reprogramming  of 
software.  For  example,  a  communications  data  table  was 
added  to  the  UPAS  by  non -programmers  after  the  software  for 
loading  the  database  and  editing  graphs/tables  was 
developed.  The  data  in  this  new  table  could  then  be 
included  in  the  creation  of  new  data  tables  and  graphs. 

Bessemer  (1996)  spoke  at  the  3rd  AAR  Conference  on  the  need 
for  system  flexibility  and  made  a  good  case  for  an  additional 
expansion  of  the  flexibility  concept  to  include  the  capability  of 
a  system  to  capitalize  on  future  circumstances  and  technical 
advances.  Future  circumstances  include  the  development  of 
training  support  packages,  the  planned  digitization  of  the 
battlefield,  and  the  fielding  of  new  weapon  systems.  Technical 
advances  might  include  the  development  of  intelligent  databases 
to  aid  in  data  analysis  and  evolving  exercise  control  systems 
(e.g,  ModSAF) . 

Internal  Consistency.  ' 

Specifying  a  particular  graphical  user  interface  (GUI)  is 
merely  one  of  the  tasks  necessary  to  ensure  that  users  of  ^  a 
system  will  not  be  frustrated  by  what  appear  to  be  inconsistent 
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procedures.  Prior  to  the  start  of  coding,  it  is  critical  that  a 
plan  be  developed  that  specifies  and  standardizes  the  way 
information  will  be  presented  to  the  user  and  the  ways  in  which 
the  user  must  respond  to  the  system.  The  plan  must  include 
provisions  for  ensuring  that  the  specifications  are  being 
followed  and  that  they  are  resulting  in  a  product  that  is 
acceptable  to  the  user.  Testing  the  product  at  various  stages  of 
development  may  lead  to  the  finding  that  sections  of  the 
procedures  need  to  be  modified  to  create  a  better  product. 

We  must  also  consider  that  the  need  to  provide  a  standardized 
interface  includes  other  software  that  the  trainer  might  use  in 
association  with  the  AAR  system.  That  is,  the  same  standard 
should  apply  to  exercise  planning  and  control  systems  and  to  any 
CBT  programs  used  to  prepare  trainers  for  employing  AAR,  exercise 
planning,  and  exercise  control  systems. 

The  STRIPES  system  provides  an  example  of  how  the  exercise 
control  and  AAR  systems  can  be  modified  to  enhance  internal 
consistency  and  reduce  operator  training  requirements.  The  plan 
view  display  in  STRIPES  was  replaced  with  that  used  in  ModSAF . 

All  the  tools  previously  available  to  trainers  when  using  ModSAF 
(e.g.,  the  capability  to  call  up  line-of -sight  displays)  are  now 
available  during  the  AAR,  and  the  same  procedures  are  used  to 
call  up  these  capabilities  during  exercises  and  during  AARs . 


Rosmarin  (1996)  pointed  out  an  important  interaction  between 
the  need  for  system  flexibility  and  that  for  internal 
consistency.  A  commonly  used  approach  to  gaining  flexibility  is 
to  use  a  mixture  of  COTS  software,  but  this  approach  can  produce 
an  overall  product  in  which  key  features  of  the  user  interface 
change  as  the  user  goes  from  one  program  to  another .  Rosmarin 
pointed  out  the  need  for  developing  a  shell  around  a  mix  of  COTS 
software  to  provide  a  consistent  interface. 


Summary 

User  frustration  with  an  AAR  system  will  increase  to  the 
extent  that  the  user  cannot  control  the  collection,  storage, 
analysis,  and  display  of  data.  Frustration  levels  will  also 
increase  to  the  extent  that  methods  for  presenting  information 
and  methods  for  controlling  the  system  differ  across  system 
functions  or  among  related  software  systems. 
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Recommendations 


Items  that  would  enhance  the  flexibility  of  the  STAARS 
recommended  for  inclusion  in  the  ORD  are  listed  below. 

o  STAARS  must  allow  users  to  create  and  modify  data  summary 
graph  and  table  options . 

o  STAARS  must  allow  users  to  integrate  observational  data 
with  electronic  data. 

o  STAARS  must  provide  the  user  with  the  capability  to  examine 
all  data  f rom| the  network  data  stream  in  a  format  that 
supports  rapid  analyses  (e.g./by  loading  all  network  data 
into  a  relational  database  management  system) . 

o  STAARS  must  offer  flexibility  in  terms  of  unit  structures 
and  the  pairing  of  individual  entities  with  units. 

o  STAARS  must  allow  users  a  reasonable  degree  of  control  over 
whether  and  how  terrain,  planning,  communications,  and 
network  data  are  displayed. 

Items  that  help  to  ensure  internal  consistency  in  terms  of  the 
human/machine  interface  are  listed  below. 

o  The  STAARS  human/machine  interface  must  be  internally 

consistent  in  terms  of  the  manner  in  which  information  is 
presented  to  the  user  and  the_manner  in  which  the  user  is 
required  to  respond  to  the  system. 

o  The  STAARS  human/machine  interface  must  be  consistent  with 
the  interfaces  of  software  systems  used  in  association  with 
STAARS  (CBT  for  users  of  STAARS  and  integrated  exercise 
planning  and  control  systems) . 

o  The  STAARS  human/machine  interface  must  be  developed 
according  to  a  plan  encompassing  continual  testing  of 
adherence  to  standards,  early  user  acceptance  testing  of 
interface  components,  and  refinements  of  standards  to 
improve  user  acceptance. 
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The  suggestions  regarding  statements  that  might  be  included 
in  the  STAARS  ORD  to  help  ensure  system  flexibility  are  presented 
in  reaction  to  specific  problems  that  have  been  encountered 
repeatedly.  We  need  a  more  proactive  approach  to  ensuring 
system  flexibility.  This  might  be  accomplished  by  developing 
more  general  standards  for  system  flexibility  and/or  by 
developing  procedures  for  early  identification  of  potential 
problems  for  specific  applications.  Cubic  Corporation  s  effort 
to  attain  a  high  degree  of  flexibility  for  the  Vision  XXI  AAR 
system  for  constructive  simulations  by  using  a  minimum  of  system- 
unique  software  and  a  maximum  of  COTS  software  is  an  example  of 
an  innovative  attempt  to  develop  a  more  general  standard  for 
flexibility  (Huff,  1996) .  Employing  task  analysis  procedures  to 
identify  potential  flexibility  issues  concerning  the  integration 
of  AAR  systems  with  exercise  planning  systems  might  make  a  good 
test  bed  for  developing  procedures  for  the  early  identification 
of  flexibility  problems. 


Preparation  of  Take  Home  Packages 


Background 

The  Take  Home  Package  (THP)  concept  involves  providing  units 
with  performance  information  that  they  can  review  after  an 
intensive  period  of  training  to  help  define  future  training 
strategies.  The  THP  can  play  at  least  three  roles.  First,  it 
can  reinforce  lessons  learned  during  AARs  for  individual 
exercises .  Second,  it  can  be  used  to  provide  feedback  regarding 
aspects  of  performance  that  were  important  but  not  addressed 
during  AARs.  Third,  the  THP  can  be  used  to  present  information 
on  trends  in  performance  across  exercises .  These  trends  might 
indicate  a  persistent  problem  in  performance  (e.g.,  the  earliest 
the  platoon  reported  contact  to  the  company  commander  was  twenty 
minutes  after  first  contact)  or  an  improvement  in  performance 
(e.g.,  the  number  of  crews  boresighting  their  weapons  prior  to 
the  start  of  the  exercise  increased  from  ten  percent  to  fifty 
percent  to  ninety  percent) .  In  order  to  support  this  last 
application  there  is  a  need  to  store  data  in  a  form  that  supports 
the  analysis  of  trends  across  exercises. 

BTAAR5  and  PIS  Standards  Guidance 

THPs  and  THP  preparation  are  not  addressed  in  the  STAARS  MNS. 
Instead,  the  THP  role  appears  to  be  replaced  by  the  input  that 
STAARS  provides  to  the  SATS  Component  of  Warfighter  XXI .  THPs 
are  not  addressed  by  the  draft  DIS  standard  for  exercise 
management  and  feedback  (Institute  for  Simulation  and  Training, 
1994)  . 

History  and  Status  of  Research 

Variables  Affecting  the  Utility  of  THPS.  The  NTC  THP  is  a 
mixed  media  package  that  includes  written  descriptions  of  unit 
performance  and  videotapes  of  AAR  sessions.  The  NTC  THP  makes 
use  of  written  descriptions  of  individual  exercises  organized  by 
battlefield  operating  systems.  These  written  materials  also 
include  descriptions  of  performance  trends  and  recommendations 
for  improving  performance . 
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Units  typically  find  it  difficult  to  make  use  of  these 
materials  due  to  personnel  turnover  after  rotations  to  the  NTC 
and  due  to  the  amount  of  time  required  to  analyze  these  data 
(Fobes  &  Meliza,  1989) .  One  would  expect  that  units 
participating  in  virtual  exercises,  constructive  exercises,  and 
live  exercises  at  home  stations  to  be  in  a  better  position  to 
make  use  of  the  THP  concept,  because  personnel  turnover  should  be 
less  of  a  problem. 

In  talking  with  unit  leaders  about  if  and  how  various 
components  of  the  NTC  THP  were  used,  Fobes  and  Meliza  (1989) 
found  that  VCR  tapes  of  AAR  sessions  were  borrowed  for  review  by 
units  preparing  to  ^o  to  the  NTC.  These  tapes  were  considered  to 
be  a  training  mediuit'  in  their  own  right . 

The  personnel  resources  required  to  prepare  NTC  THPs  go 
beyond  those  available  in  many  training  situations.  A  number  of 
efforts  have  worked  toward  the  development  of  electronic  THPs 
to  help  reduce  the  time  and  effort  required  to  prepare  these 
products.  The  intent  of  these  efforts  is  to  provide  THPs  on 
media  that  are  readily  available  to  units,  such  as  VCR  tape 
players  and  personal  computers . 

The  ATAFS  loads  AAR  displays  to  VCR  tape.  The  trainer  may 
load  only  the  displays  used  for  an  AAR,  or  he/she  may  add 
additional  displays  not  used  during  the  AAR.  The  trainer  may 
also  choose  to  load  the  entire  exercise  to  VCR  tape.  A  drawback 
in  this  approach  is  that  it  does  not  provide  information  about 
what  exercise  participants  said  during  the  AAR.  Information 
about  what  the  unit  identified  as  the  source  of  unit  strengths 
and  weaknesses  and  corrective  actions  are  not  included  in  the 
THP. 


The  electronic  THP  concept  might  be  modified  to  include 
information  about  the  causes  of  strengths  and  weaknesses  and 
corrective  actions.  One  approach  is  to  tape  AAR  sessions. 

Another  approach  might  be  for  a' trainer  to  summarize  the  outcomes 
of  the  AAR  in  an  electronic  format  using  brief  statements  or 
responses  to  a  checklist  of  items. 

The  CATT  Training  Exercise  Development  System  (CATT-TREDS) 
project  includes  the  collection  of  data  on  unit  performance  at 
the  MTP  subtask  standard  level  and  the  application  of  these  data 
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in  planning  future  exercises*  CATT-TREDS  is  being  tried  out 
within  a  number  of  units,  and  this  project  should  provide  us  with 
information  regarding  the  feasibility  of  collecting  detailed  unit 
performance  evaluations  on  a  regular  basis. 

There  are  many  questions  about  the  intended  use  of  THPs  that 
need  to  be  addressed  before  we  can  specify  the  requirements  for  a 
STAARS.  At  least  a  portion  of  these  questions  need  to  be 
addressed  by  representatives  of  the  SATS  component  of  Warfighter 
XXI  regarding  the  information  needs  of  SATS.  For  example,  should 
a  THP  address  each  separate  exercise,  only  the  last  exercise  of  a 
particular  type,  and/or  trends  in  performance  across  exercises? 

The  THP  Preparation  Process.  The  work  involved  in  preparing 
a  THP  can  be  illustrated  by  considering  what  0/Cs  at  the  Fort 
Knox  VTP  do  to  prepare  THPs.  Again,  remember  that  one  of  the 
reasons  for  preparing  THPs  is  that  not  enough  time  is  available 
during  AARs  to  cover  every  critical  aspect  of  unit  performance. 

During  exercises,  O/Cs  attempt  to  complete  a  form  containing 
relevant  performance  standards  from  MTP  documents .  The  RCVTP  THP 
is  expected  to  including  ratings  on  these  standards  and  to  give 
greater  weight  to  ratings  given  later  rather  than  earlier  in 
training  a  specific  MTP  task.  Along  with  these  ratings,  the  0/Cs 
provide  narrative  descriptions  that  may  explain  ratings  and/or 
provide  guidance  for  improving  performance  in  the  future.  This 
guidance  might  involve  references  to  doctrinal  literature  or 
other  types  of  guidance  falling  under  the  "a  way"  concept. 

Making  it  Easier  to  Relate  Performance  Outcomes  to  Corr^gfiv^ 
Actions .  The  large  number  of  MTP  performance  standards  that 
might  be  applied  to  exercises  reduces  the  possibility  that  a 
thorough  rating  of  a  unit  on  every  standard  will  be  provided  as 
input  to  the  SATS  components  of  Warfighter  XXI.  For  example,  the 
MTP  for  a  Battalion  Task  Force  covers  1,489  subtask  standards 
distributed  among  343  subtasks.  A  general  approach  to  reducing 
the  data  collection  load  is  to  provide  ratings  of  unit 
performance  at  levels  higher  than  the  subtask  standard  level. 

For  example,  observer/controllers  within  the  Reserve  Component 
Virtual  Training  Program  at  Fort  Knox,  KY,  rate  units  at  the 
subtask  level  in  terms  of  whether  they  need  to  "train  to  improve" 
versus  "train  to  sustain"  on  specific  subtasks  (Shlechter,  et 
al. ,  1995) . 
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Anothsir  approach  to  using  a  mission  structure  that  provides  a 
more  succinct  description  of  unit  performance  is  the  use  of 
critical  combat  functions  (CCFs) .  CCFs  are  higher  than  task 
level  but  lower  than  the  battlefield  operating  system  (BOS)  level 
(BDM/  in  preparation) .  Another  similar  approach  taken  under 
STRICOM's  Advanced  Distributed  Simulation  Technology  contract 
with  Loral  Advanced  Distributed  Simulation  is  the  development  of 
a  Training  Architecture  Database  in  which  MTP  tasks  are 
referenced  to  functional  areas  under  the  BOS  framework. 

A  fourth  approach  to  reducing  the  data  load  for  satisfying 
SATS  requirements  is  to  summarize  data  in  terms  of  performance 
trends  for  a  particular  unit.  For  example,  if  a  unit  repeatedly 
fails  to  execute  a  combat  task  across  missions,  or  repeatedly  has 
problems  in  performing  the  same  type  of  activity  within  a  mission 
(a  unit  fails  to  communicate  with  sister  units  during  each  phase 
of  a  mission) ,  then  this  information  would  have  a  high  payoff  for 
inputting  into  the  SATS. 

Summary 

The  need  for  THPs  after  virtual  training  exercises  may  be 
greater  than  that  for  capstone  training  events  like  an  NTC 
rotation  for  the  simple  reason  that  units  are  more  likely  to  have 
an  opportunity  to  take  advantage  of  the  information  provided. 

It  must  also  be  considered  that  we  are  moving  into  a  new  era 
where  a  portion  of  the  information  provided  in  THPs  might  need  to 
be  provided  in  a  format  that  can  be  used  by  an  automated  training 
management  system  like  SATS.  _ 

In  addition  to  providing  a  THP  product  that  supports  SATS, 
there  are  other  goals  to  keep  in  mind  when  considering  the 
requirements  for  THP  preparation.  We  need  to  move  away  from 
paper-based  THPS  toward  electronic  THPs .  We  also  need  to  make  it 
easier  for  the  preparers  of  THPs  to  relate  performance  outcomes 
to  corrective  actions .  Addressing  this  latter  goal  involves 
exploring  the  application  of  tafek  structures  and  techniques 
(e.g.,  looking  for  repeating  patterns)  that  make  it  easier  to 
organize  and  summarize  feedback. 
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Recommendations 


The  STAARS  should  be  capable  of  changing  the  task  structure 
used  to  support  assessments  of  unit  performance  (6.g./  MTP 
structure  versus  CCF  structure)  without  reprogramming  of 
software . 

Research/development  issues  that  need  to  be  addressed  are 
listed  below. 

o  We  need  to  find  out  whether  there  are  benefits  to  providing 
a  THP  separate  from  those  gained  by  loading  unit 
performance  evaluations  into  a  SATS. 

o  We  need  to  know  how  to  use  STAARS  to  support  SATS  functions 
without  imposing  a  heavy  burden  on  the  shoulders  of 
trainers  by  finding  out  the  minimum  type  and  level  of  data 
input  required  to  support  effective  training  strategy 
development . 


\ 


55 


Providing  Input  for  the  Army  Training  Digital  Library 


Background 


The  intent  of  changes  in  doctrine,  training,  leader 
development,  organization,  materiel,  and  soldier  systems  (DTLOMS) 
is  to  improve  performance  on  unit  collective  tasks .  Data  from 
collective  training  exercises  might  make  significant 
contributions  to  Research  Development  and  Acquisition  (RDA) and 
Advanced  Concepts  and  Requirements  (ACR)  efforts. 

.qTAARS  and  PIS  Standards  Guidance 

The  STAARS  MNS  specifies  that  "STAARS  standardizes  the 
automated  storage,  distribution,  and  retrieval  of  AAR  data  within 
the  ATDL  architecture."  The  MNS  also  specifies  that  STAARS  will 
provide  information  through  the  ATDL  to:  the  training,  exercises 
and  military  operations  (TEMO)  community;  the  research, 
development,  and  acquisition  (RDA)  community;  and  the  advanced 
concepts  and  requirements  (ACR)  community. 

The  draft  standard  for  exercise  management  and  feedback  does 
not  specifically  address  the  issue  of  archiving  exercise  data  for 
purposes  (Institute  for  Simulation  and  Training,  1994)  , 
however,  certain  capabilities  noted  in  the  draft  are  relevant  to 
archiving  for  research  applications.  Namely,  the  data  should  be 
archived  in  a  manner  that  allows  reproduction  of  the  data  stream 
exactly  as  received  and  supports  analysis  across  exercises. 

History  and  Status  of  Research 

Non- attribution .  A  major  concern  expressed  during  the  second 
AAR  conference  was  protecting  the  identity  of  units  when  their 
performance  data  are  loaded  into  a  research  database  like  the ^ 
ATDL.  The  developers  of  STAARS  and  the  ATDL  must  jointly  decide 
whether  unit  IDs  are  to  be  removed  before  data  are  made  available 
to  researchers,  or  if  researchets  are  to  be  made  responsible  for 
maintaining  confidentiality. 

Protection  of  unit  identities  has  long  been  a  policy  of  ARI 
and  the  TRADOC  Center  for  Lessons  Learned  (CALL)  regarding  the 
analysis  of  data  from  unit  rotations  to  CTCs.  In  this  case,  unit 
IDs  are  contained  with  the  raw  data  and  there  are  multiple  checks 
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to  insure  that  specific  units  cannot  be  identified  in  the 
processed  data.  For  example,  ARI,  CALL,  and  the  NTC  Operations 
Group  might  separately  check  to  see  that  confidentiality  is 
maintained  for  a  report.  Use  of  the  ATDL  may  be  less  centralized 
than  use  of  the  current  CTC  data  archive  and  require  a  greater 
degree  of  ID  removal  before  data  are  given  to  researchers. 

ThP  to  Archive  Planning.  Communicatigns . 

AHmini  strati Vf>  Data  Along  with  Other  TVP.aS;  One  of  the  key 

concerns  of  researchers  is  documenting  the  conditions  under  which 
data  were  collected  to  make  sure  data  are  aggregated  only  when 
they  are  comparable.  The  actions  of  a  unit  must  often  be 
interpreted  in  terms  of  the  complete  METT-T  situation  that 
confronts  a  unit  (Karins,  Atwood,  &  Root,  1990) .  Unfortunately 
there  is  a  tendency  to  overestimate  the  capability  of  an 
electronic  battlefield  to  support  research.  One  assumes  the 
capability  to  capture  detailed  data  from  the  network  (e.g.,  the 
direction  in  which  gun  tubes  are  pointed)  guarantees  the 
capability  to  examine  these  data  and  draw  meaningful  conclusions. 
However,  users  of  such  data  rapidly  discover  how  difficult  it  is 
to  interpret  data  without  information  about  the  unit's  plan  for 
the  mission,  radio  communications,  and  terrain  features.  To 
support  research,  planning  data  and  communications  data  must  be 
archived  along  with  network  data . 

A  lesson  learned  from  the  Multi-Service  Distributed  Training 
Testbed  {MDT2)  Project  is  that  it  is  crucial  to  collect 
administrative  data  to  support  feedback  and  research  applications 
of  exercises.  For  example,  the  senior  MDT2  trainer  reduced  the 
volume  of  artillery  fire  support  missions  requested  by  a  unit  to 
help  keep  the  network  data  load  from  becoming  too  large.  A 
researcher  might  reach  erroneous  conclusions  by  assuming  that  the 
volume  of  artillery  fires  directly  reflects  unit  requests. 

In  another  case  during  MDT2  data  collection,  the  senior 
trainer  used  a  "bomb  button"  to  destroy  an  enemy  vehicle  targeted 
by  an  aircraft  performing  a  CAS* mission.  The  "bomb  button'  is 
frequently  used  in  SIMNET  exercises  to  remove  entities  from 
exercises  by  impacting  a  500  pound  bomb  on  the  vehicle.  The 
trainer  decided  that  the  pilot  was  doing  everything  right  and  the 
original  bomb  should  have  resulted  in  damage  or  destruction  of 


57 


the  vehicle.  The  problems  analyzing  the  data  stream  after  this 
exercise  begin  when  one  realizes  that  the  number  of  bomb  releases 
is  less  than  the  number  of  impacts. 

The  Need  to  Summarize  the  Contents  of  an  Exgrcigg-  Another 
concern  of  researchers  is  that  of  reducing  the  amount  of  data 
that  need  to  be  examined.  Again,  the  capability  to  operate  and 
collect  data  in  an  electronic  format  deludes  researchers  into 
believing  that  these  data  can  be  examined  quickly.  This  is  true 
only  if  one  does  not  have  to  rely  on  a  complete  replay  of  the 
data  stream  to  compare  one  exercise  with  another .  To  the  extent 
that  data  analysis  tools  are  available  to  remove  the  need  for 
complete  replays,  ths  electronic  data  format  can  be  beneficial. 

In  a  recent  meeting  on  AAR  capabilities,  the  Project  Manager 
for  Combined  Arms  Tactical  Trainer  (PM-CATT)  expressed  the  need 
for  an  archiving  system  that  provides  a  summary  of  exercise 
events  that  supports  analyses  of  data  across  exercises.  As  he 
pointed  out,  trying  to  analyze  data  by  replaying  entire  exercises 
is  just  too  cumbersome  and  time-consuming. 

.qtandardizina  PIS  Data  Loaapr  File  Formats.  ARI,  and  other 
organizations,  have  encountered  problems  in  trying  to  use 
archived  data  from  SIMNET  exercises  because  multiple  archiving 
formats  are  in  use.  The  Institute  for  Defense  Analysis  (IDA)  and 
STRICOM  recognized  the  need  for  standardizing  the  format  of  DIS 
data  logger  files  and  a  working  group  led  by  STRICOM  developed  a 
standardized  file  format  (Garnsey,  et  al.,  1995). 

In  addition  to  standardizing  the  format  of  data  files,  this 
effort  also  started  the  job  of  including  non-network  data 
necessary  to  interpret  network  data  in  the  logger  file.  Examples 
of  non-network  data  include  the  identification  of  the  terrain 
database  used  for  the  exercise,  the  type  of  mission  being 
executed,  the  unit's  plans  for  the  operation,  and  unit  map 
overlays  and  graphics  (e.g.,  showing  the  locations  of  control 
measures) . 

A  first  draft  of  the  standard  file  format  is  available 
through  the  Tactical  Warfare  Simulation  and  Technology 
Information  Analysis  Center  (TWISTIAC) .  The  process  of  testing 
and  refining  the  standard  data  logger  format  is  expected  to 
continue  in  the  immediate  future.  From  the  perspective  of  human 
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performance  analysis,  two  of  the  major  issues  to  be  addressed  are 
whether  the  format  includes  all  of  the  types  of  non-network  data 
required  to  support  performance  analysis  and  the  amount  and 
variety  of  network  data  collected  is  excessive . 

An  important  lesson  learned  from  the  UPAS  project  has  been 
that  the  entire  PDU  stream  is  not  necessary  to  support^ 
performance  analysis .  The  high  frequency  at  which  entity  status 
data  must  be  sent  over  the  network  to  support  a  simulation  goes 
well  beyond  what  is  required  for  analysis.  Time  sampling 
techniques  can  be  used  to  reduce  the  size  of  exercise  data  files. 

The  need  for  an; archiving  system  that  increases  the 
efficiency  of  reseairch  can  be  met  to  an  extent  by  using  a 
strategy  developed  within  the  UPAS  project.  The  UPAS  employs  two 
databases,  a  logger  file  capable  of  driving  exercise  replays  and 
a  set  of  relational  database  tables  used  to  support  data 
analyses.  Menus  of  data  summary  table  and  graph  options  are 
integrated  with  the  relational  database  to  support  analyses,  and 
the  menus  can  be  modified  or  expanded  by  users.  For  archiving 
purposes,  only  the  logger  file  needs  to  be  saved.  The  relational 
data  tables  can  be  quickly  generated  from  the  logger  file  to 
support  analyses  using  the  menus  of  table  and  graph  options. 
Having  exercise  data  in  relational  data  tables  makes  it  possible 
to  combine  exercise  data  across  exercises  in  a  format  that 
supports  analyses  across  exercises. 

Archiving  of  Performance  Measures.  At  the  present  time  there 
is  no  centralized  source  of  information  regarding  measures  of 
performance  that  have  been  used  in  the  DIS  environment .  BDM 
Corporation,  under  the  sponsorship  of  ARI,  undertook  a  brief 
effort  to  define  the  concept  for  a  DIS  Taxonomy  of  On  and  Off- 
Line  (DIS-TOOL)  Performance  Variables  (Winsch,  Clifton,  Sc.  Atwood, 
in  preparation) .  As  the  name  suggests,  this  taxonomy  was 
intended  to  deal  with  non-network  data  (off-line)  as  well  as 
network  data  (on-line) . 

The  general  concept  called  for  archiving  of  measures  of 
performance  (MOP)  and  historical  data  on  the  use  of  the  MOP.  One 
of  the  major  goals  of  the  taxonomic  system  was  to  find  a  way  of 
structuring  the  archive  in  a  way  to  help  a  wide  variety  of  users 
(combat  developers,  computer  generated  force  developers,  materiel 
developers,  training  developers,  and  operational  test  and 
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evaluation  developers)  to  identify  MOP  relevant  to  their 
objectives.  Plans  called  for  using  advocates  for  each  potential 
user  group  in  the  development  of  the  taxonomy  to  insure  the 
structure  and  terminology  used  for  the  system  fit  each  group. 

This  effort  was  terminated  due  to  a  lack  of  funding  in  the 
early  stages  of  taxonomy  development .  The  need  for  the  product 
has  not  diminished. 

Summary 


The  requirement  to  provide  data  for  the  ATDL  might  make  it 
necessary  to  impose;  certain  features  on  the  STAARS  in  order  to 
safeguard  the  identity  of  units.  Then  again,  responsibility  for 
safeguarding  this  information  might  reside  with  the  ATDL 
component . 

In  order  to  interpret  the  data  stream  collected  by  STAARS  in 
the  context  of  research  applications  it  will  be  necessary  to 
supplement  the  network  data  with  unit  planning  data,  information 
about  the  computer  generated  force  used  to  support  the  exercise, 
and  data  on  trainer/exercise  director  interventions  during  the 
exercise.  All  three  training  environments  have  been  used  to 
support  combat  developments  testing  in  the  past,  so  there  are 
groups  of  researchers  we  can  call  upon  to  identify  information 
that  must  be  added  to  network  data  streams  to  support  research 
applications.  STAARS  developers  should  also  provide  input  to 
ongoing  efforts  to  standardize  logger  file  formats.  Efforts  to 
identify  the  information  needed  to  support  research  applications 
of  STAARS  data  would  be  facilitated  if  we  had  an  archive  of 
measures  of  unit  performance  used  in  the  three  training 
environments,  but  no  such  archive  exists. 

Recommendations 

Add  the  capabilities  described  below  to  the  STAARS  ORD. 

I 

o  The  system  must  suppprt  easy  removal  of  unit  and  simulator 
identifier  data. 

o  The  system  must  support  the  recording  of  information  about 
trainer  or  work  station  operator  interventions  during 
exercises  (e.g.,  a  trainer  or  operator  uses  a  bomb  button 
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to  destroy  a  threat  to  a  unit  so  the  unit  can  continue 
executing  its  mission)  in  a  manner  that  allows  these 
interventions  to  be  correlated  in  time  with  other  exercise 
events . 

The  system  must  support  the  recording  of  planning  data  and 
observational  data  in  a  manner  that  allows  these  data  to  be 
correlated  in  time  with  other  exercise  events. 


Interoperability  Concerns 


Background 


The  Workshops  on  Standards  for  the  Interoperability  of 
Distributed  Simulations  are  concerned  with  specifying  the 
capabilities  a  system  must  have  to  interoperate  with  other 
systems  in  the  DIS  environment.  However,  compliance  with  the 
standards  produced  through  these  workshops  is  only  part  of  the 
requirements  that  must  be  met  to  make  sure  simulation  systems  can 
effectively  share  the  same  AAR  system. 

The  data  packets  a  system  must  be  able  to  produce  and  react 
to  in  supporting  an’ interactive  simulation  are  required  for  a 
system  to  be  DIS  compliant.  There  are  other  data  packets  falling 
under  the  rubric  of  "simulation  management"  that  are  optional 
rather  than  being  required  (IEEE,  1993) .  Many  of  the  data 
packets  falling  in  this  class  were  developed  specifically  to 
support  performance  measurement  applications,  such  as  AARS . 
Further,  systems  are  free  to  vary  in  terms  of  the  manner  in  which 
they  convey  information  about  a  specific  event.  'Using  an  AAR 
system  to  collect'  data  from  across  a  mixture  of  simulators  will 
require  a  greater  degree  of  standardization  than  that  required 
for  DIS  compliance. 

STAARS  and  DIS  Standards  Guidance 


The  STAARS  MNS  requires  that  the  STAARS  support  training  with 
all  collective  training  systems,  but  the  MNS  does  not  specify 
what  is  required  to  operate  effectively  across  systems.  On  the 
other  hand,  the  primary  purpose  of  the  DIS  Standards  Conferences 
is  to  promote  interoperability. 

Status  of  Research 


Data  Communication  Protocols.  CCTT  will  be  the  first  system 
to  be  fielded  as  part  of  the  Artny' s  Combined  Arms  Tactical 
Trainer  (CATT)  family,  and, it  will  employ  DIS  protocols  rather 
than  SIMNET  protocols.  Future  members  of  the  CATT  family  must  be 
interoperable  with  CCTT  to  allow  the  linking  of  CATT  systems  in  a 
single  exercise.  CCTT  must  also  be  interoperable  with  SIMNET  to 
allow  SIMNET  and  CCTT  to  be  linked  in  common  exercises  in  the 
near  future  (U.S.  Army  Armor  School,  1990) . 
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One  of  the  earliest  problems  implementing  DIS  protocols 
concerned  the  loss  of  a  direct  replacement  for  the  SIMNET  Change 
of  Status  PDU.  Under  SIMNET,  this  PDU  was  generated  by  an  entity 
when  it  went  through  a  major  change  in  status  (damaged, 
destroyed,  reincarnated) ,  and  it  conveyed  information  about  the 
nature  of  the  status  change,  the  cause  of  the  status  change,  and 
specific  entities  responsible  for  the  change  (Pope  &  Schaffer, 
1991) .  The  data  from  this  PDU  could  be  used,  for  example,  to 
find  out  that  BLUFOR  tanlc  All  was  destroyed  by  fire  from  REDFOR 
tanJc  B12  at  16:33:10.  Without  this  PDU,  the  data  analysis  system 
must  recreate  events  to  find  out  why  a  status  change  occurred  and 
who  was  responsible  for  the  change  (Meliza,  1995) .  This  gap  in 
the  data  stream  was;  one  of  several  identified  when  attempting  to 
produce  exercise  data  summaries  specified  in  the  CCTT 
reguirements  document  (Lacy,  Tuttle,  &  Meliza,  1994) . 

The  solution  to  the  information  gap  associated  with  the  loss 
of  the  Status  Change  PDU  was  to  implement  a  version  of  the  DIS 
Event  Report  PDU  that  replaces  the  Status  Change  PDU.  This 
implementation  is  described  in  the  CCTT  Interoperability 
Description  Document  (Sherilcon,  1995)  .  In  order  for  other 
training  devices  to  link  with  CCTT  in  common  exercises  where  AAR 
aids  cover  damage  inflicted  by  all  players,  these  other  devices 
must  be  programmed  to  generate  the  same  version  of  the  Event 
Report  PDU  generated  by  CCTT. 

We  can  expect  additional  problems  and  solutions  to  emerge 
regarding  communication  protocols  over  the  next  several  years, 
due  to  the  fact  that  we  are  continually  making  the  DIS 
environment  more  realistic  by  adding  new  entities  and  expanding 
the  environmental  models  used  in  the  DIS  environment.  We  will 
again  find  that  some  of  the  aspects  of  behavior  we  want  to 
measure  cannot  be  measured  easily  without  changes  in  the  network 
data  stream. 

T.^T-raiTi  Databases  and  Other  Environmental  Models.  •  SIMNET  was 
very  primitive  in  terms  of  the  t>lay  of  environmental  variables 
(e.g.,  day/night,  wind,  rain,  electromagnetic  spectrum).  Over 
time,  we  are  adding  the  capability  to  provide  greater  variation 
in  environmental  variables  and  entities  that  can  respond  to  these 
variations.  The  capability  of  any  one  training  device  to 
realistically  play  all  environmental  variables  is  restricted  by 
limits  in  processing  power,  so  most  training  devices  are  likely 
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to  emphasize  those  environmental  variables  considered  most 
critical  to  the  device  proponent.  For  example/  ground  forces 
will  want  detailed  play  of  terrain  variables,  while  air  forces 
will  want  detailed  play  of  the  electromagnetic  spectrum. 

To  properly  interpret  exercise  data,  one  must  have 
information  about  the  sensitivity  of  entities  to  the  various 
environmental  models  and  information  about  the  detail  of  these 
models.  The  job  of  collecting  information  grows  in  complexity  as 
individual  training  devices  become  more  sophisticated  and  as  the 
jnix  of  training  devices  included  in  an  exercise  become  more 
heterogenous.  Very  little  progress  has  been  made  regarding 
methods  for  collecting  and  interpreting  information  about ^ 
environmental  model's  except  for  terrain  database  correlation 
(Spuhl  Sc  Findley,  1994)  . 

The  Chemical,  Biological,  and  Radiological  (CBR)  Models  and 
Simulation  Consortium  is  expanding  terrain  models  and  entity 
behaviors  to  address  CBR  contamination  to  support  DIS 
applications  to  combat  developments  testing.  This  group  plans  to 
use  STRIPES  as  an  AAR  system,  and  it  expects  to  modify  STRIPES 
source  code  to  support  performance  measurement  in  a  more  complex 
battle  space.  The  consortium  includes  the  U.S.  Army  Chemical 
School,  Natick  Research  Development  and  Engineering  Center,  the 
U.S.  Army  Nuclear  and  Chemical  Agency,  the  Ballistic  Missile 
Defense  Organization,  U.S.  Army  Research  Laboratory,  PM  Smoke  and 
Obscurants,  PD  Bio  Defense,  PM  NBC  Defence,  TRADOC  Analysis 
Command,  Naval  Surface  Warfare  Center,  Institute  for  Defense 
Analysis,  Defence  Nuclear  Agency,  and  the  Dismounted  Battlespace 
Battlelab. 

Computer  Oenerated  Forces  (CGF),.  As  previously  mentioned  in 
this  report,  memories  and  knowledge  of  exercise  participants  are 
important  sources  of  information  for  AARs.  The  memories/ 
knowledge  that  are  important  includes  those  from  sister  units 
with  which  the  unit  works  in  performing  collective  tasks,  and  it 
includes  information  from  the  opposition  force.  This  is  true 
whether  the  sister  unit  and  opposition  force  are  composed  of 
manned  units  or  CGF. 

Information  about  when  a  CGF  detects,  identifies,  and 
recognizes  another  force  is  not  broadcast  over  the  network  so^ 
that  it  can  be  picked  up  by  an  AAR  system.  This  information  is 
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retained  in  the  internal  database  of  at  least  some  CGFs .  These 
data  might  be  loaded  directly  from  the  CGF  to  the  AAR  system, 
but,  because  each  CGF  differs  in  terms  of  the  manner  in  which 
these  data  are  stored  internally,  this  approach  would  require 
software  unique  to  each  type  of  CGF.  Alternatively,  CGFs  might 
be  required  to  send  out  the  required  information  over  the 
network,  using  an  Event  Report  PDU  or  similar  DIS  simulation 
management  PDU.  There  is  a  need  for  the  Army  to  decide  what 
information  it  needs  from  CGFs  and  select  the  most  efficient 
means  of  collecting  that  data  for  use  on  an  AAR  system. 

Proper  interpretation  of  exercise  data  in  which  a  CGF  is 
involved,  especially  for  test  purposes,  requires  information 
about  the  rules  controlling  the  behavior  of  the  CGF.  These  rules 
will  vary  from  one  CGF  to  another.  For  example,  in  comparing 
sensitivity  to  terrain  features  of  ModSAF  Version  1.2  with  SAFOR 
Version  4.3.3,  Meliza  and  Vaden  (1995)  found  the  speed  of  ModSAF 
entities  varied  as  a  function  of  terrain,  while  the  speed  of 
SAFOR  entities  was  largely  insensitive  to  terrain.  In  a  case 
where  a  ModSAF  and  SAFOR  vehicles  moved  over  the  same  terrain 
with  inclines  and  declines  of  as  much  as  40  percent,  the  SAFOR 
vehicle  spent  98  percent  of  the  time  traveling  at  a  speed  of  29 
km/h.  The  ModSAF  entity  demonstrated  the  distribution  of  speeds 
shown  in  Table  1 . 
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Table  3.  %  of  Time  MODSAF  Tanks  Spent  Traveling  at  Various 

Speeds 


Speed 

(Km/Hr) 

%  of  Time 
Moving  at 

This  Speed 

Speed 
(Km/ Hr) 

%  of  Time 
Moving  at 
This  Speed 

1 

1% 

26 

1% 

3 

1% 

27 

1% 

8 

1% 

28 

1% 

9 

1% 

29 

1% 

10 

1% 

30 

1% 

12 

1% 

32 

1% 

13 

1% 

33 

1% 

14 

2% 

35 

2% 

15 

1% 

36 

54% 

16 

6% 

37 

2% 

17 

2% 

38 

1% 

18 

1% 

39 

1% 

19 

1% 

50 

1% 

20 

1% 

62 

1% 

22 

1% 

73 

1% 

24 

1% 

76 

1% 

25 

2% 

28 

1% 

Summary 

The  standards  being  developed  within  the  Workshops  on 
Standards  for  the  Interoperability  of  Distributed  Simulations  are 
a  starting  point  for  addressing  interoperability  issues  relevant 
to  AAR  systems.  The  major  benefit  derived  from  the  standard  is 
to  ensure  commonality  in  terms  of  the  generation  and 
interpretation  of  network  data  used  to  directly  support 
simulations  (e.g.,  entity  PDUs)'.  Benefits  are  also  gained  by 
providing  a  common  framework  for  the  simulation  management  PDUs 
carrying  data  relevant  to  performance  measurement,  but  to  take 
advantage  of  this  framework  requires  further  standardization  in 
the  way  these  PDUs  are  implemented.  For  Army  systems,  the 
specific  implementations  required  to  ensure  interoperability  are 
defined  in  the  CCTT  interoperability  document  (Sherikon,  1995) . 
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In  addition  to  standardizing  the  network  data  stream, 
attention  must  be  paid  to  comparing  the  environmental  models  used 
by  various  simulations,  such  as  the  terrain  databases.  Attention 
must  also  be  paid  to  comparing  the  sensitivity  of  systems, 
including  various  computer  generated  forces ,  to  environmental  and 
network  variables . 

Recommendations 

The  STAARS  ORD  should  require  that  any  AAR  system  expected  to 
use  the  DIS  data  stream  be  capable  of  interpreting  the  CCTT 
implementations  of  the  simulation  management  PDUs. 

At  the  present  time  it  is  difficult  to  state  what  an  AAR 
system  must  be  capable  of  doing  to  take  advantage  of  information 
from  CGFs .  Research  is  required  to  identify  the  best  methods  for 
collecting  what  is  currently  non-network  data  from  CGFs  to 
support  unit  performance  measurement  for  AAR,  THP,  and  research 
applications. 


l 
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Appendix  A:  Acronyms 


AAR 

ARI 

ARNG 

ATDL 

ATAFS 

ATAFS-AT 

CBS 

CGF 

CALL 

CAS 

CATT 

CBT 

CCTT 

COTS 

CTC 

DARPA 

DIS 

ENDEX 

IDA 

LD 


After  Action  Review 

Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

Army  National  Guard 

Army  Training  Digital  Library 

Automated;  Training  Analysis  and  Feedback  System 

ATAFS  Authoring  Tool 

Corps  Battle  Simulation 

Computer  Generated  Force 

Center  for  Army  Lessons  Learned 

Close  Air  Support 

Combined  Arms  Tactial  Trainer 

Computer-Based  Training 

Close  Combat  Tactical  Trainer 

Commercial  Off  the  Shelf 

Combat  Training  Center 

Defense  Advanced  Research  Projects  Agency 
Distributed  Interactive  Simulation 
End  of  Exercise  ‘ 

Institute  for  Defense  Analysis 
Line  of  Departure 
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METT-T  Mission,  Enemy,  Terrain,  Troop,  and  Time 
MNS  Mission  Needs  Statement 

ModSAF  Modular  Semi -Automated  Force 

MILES  Multiple  Integrated  Laser  Engagement  System 

MTP  Mission  Training  Plan 

MDT2  Multi-Service  Distributed  Training  Testbed 

MWTSC  Mounted  Warfare  Training  Simulation  Center  (MWTSC) 

NSC  National  Simulation  Center 

NTC  National  Training  Center 

0/C  Observer/Controller 

ORD  Operational  Requirements  Document 

PDU  Protocol  Data  Unit 

POI  Program  of  Instruction 

RCVTP  Reserve  Component  Virtual  Training  Program 

REALTRAIN  Realistic  Training 

SAFOR  Semi -automated  Force 

SATS  Standard  Army  Training  System 

SCOPES  Squad  Combat  Operations  Engagement  Simulation 
SIMITAR  Simulation  in  Training  for  Advanced  Readiness 

f 

SIMNET  Simulation  Networking 

SOP  Standard  Operating  Procedures 
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STAARS 

STOW 

STRICOM 

STRIPES 


Standard  Army  After  Action  Review  System 
Synthetic  Theater  of  War 

Simulation,  Training  and  Instrumentation  Command 

Simulation  Training  Integrated  Performance  Evaluation 
System 


TADSS 

TES 

TOM 

THP 

TRADOC 

TSP 

TTP 

TWIST  I  AC 

UPAS 


Training  Aids,  Devices,  Simulations,  and  Simulators 

Tactical  Engagement  Simulation 

Teamwork  Observation  Measure 

Take  Home  Package 

Training  and  Doctrine  Command 

Training  Support  Package 

Tactics,  Techniques,  and  Procedures 

Tactical  Warfare  Simulation  and  Technology  Information 
Analysis  Center 

Unit  Performance  Assessment  System 


l 


79 


